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^^Accountability  *  Integrity  *  Reliability 


United  States  General  Accounting  Office 
Washington,  D.C.  20548 


December  18,  2000 

The  HonorableJ  ohn  Warner 
Chairman,  Committee  on  Armed  Services 
United  States  Senate 

Dear  Mr.  Chairman: 

Because  high  performance  computing  is  an  important  enabling  technology 
for  military  purposes,  the  U.S.  government  controls  the  export  of  high 
performance  computers  to  sensitive  destinations,  such  as  Russia  and 
China,  based  on  foreign  policy  and  national  security  concerns.  A  high 
performance  computer1  has  both  civilian  and  military  applications,2 
operates  at  or  above  a  defined  performance  threshold,  and  requires  an 
export  license  to  particular  destinations,  according  to  the  Commerce 
Department,  the  agency  responsible  for  licensing  dual-use  items.  U  .S. 
policy  with  respect  to  the  export  of  sensitive  technology,  including 
computers,  is  to  seek  a  balance  between  the  U  .S.  economic  interest  in 
promoting  exports  and  its  national  security  interests  in  both  maintaining  a 
military  advantage  over  potential  adversaries  and  denying  the  spread  of 
technologies  used  in  developing  weapons  of  mass  destruction. 

The  President  has  periodically  changed  the  performance  thresholds  above 
which  licenses  are  required  based  on  technological  advances.  The  National 
Defense  Authorization  Act  of  1998  requires  the  President  to  provide  a 
report  to  Congress  justifying  each  change  for  exports  of  high  performance 
computers  to  certain  sensitive  countries.  It  requires  the  report,  at  a 
minimum,  to  (1)  address  the  extent  to  which  high  performance  computers 
with  capabilities  between  the  established  level  and  the  new  proposed  level 
of  performance  are  available  from  other  countries,  (2)  address  all  potential 
uses  of  military  significance  to  which  high  performance  computers  at  the 


1  High  performance  computers  are  regulated  based  on  their  composite  theoretical 
performance  as  measured  in  millions  of  theoretical  operations  per  second  (MTOPS). 

2  An  application  is  a  program  or  group  of  programs  designed  for  end  users.  Software  can  be 
divided  into  two  general  classes:  systems  software  and  applications  software.  Systems 
software  consists  of  low-level  programs  that  interact  with  the  computer  at  a  very  basic  level, 
including  operating  systems,  compilers,  and  utilities  for  managing  computer  resources.  In 
contrast,  applications  software  (also  called  end-user  programs)  includes  database 
programs,  word  processors,  and  spreadsheets.  Figuratively  speaking,  applications  software 
sits  on  top  of  systems  software  because  it  is  unable  to  run  without  the  operating  system  and 
system  utilities. 
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new  levels  could  be  applied,  and  (3)  assess  the  impact  of  such  uses  on  U.S. 
national  security  interests.  The  President  announced  in  August  2000  that 
the  control  threshold  above  which  computers  exported  to  countries  like 
Russia  and  China  would  need  a  license  would  be  28,000  millions  of 
theoretical  operations  per  second  (MTOPS)  effective  immediately. 

Rapid  technological  advances  in  computing  power  have  fueled  concerns 
over  the  continued  effectiveness  of  current  computer  export  controls.  You 
raised  concerns  about  whether  the  computer  export  licensing  process 
continues  to  protect  U  .S.  national  security  interests.  As  agreed  with  your 
office,  this  report  discusses  (1)  the  effectiveness  of  current  export  controls 
in  preventing  countries  of  concern  from  obtaining  high  performance 
computing  capabilities  that  can  be  used  for  military  applications  and 
(2)  our  evaluation  of  the  President's  justifications  for  changing  the 
computer  control  thresholds.  You  also  asked  us  to  identify  possible  ways  of 
protecting  U  .S.  national  security  interests  related  to  exports  of  high 
performance  computers.  We  obtained  the  views  of  government  and  private 
experts  on  this  matter,  and  they  are  presented  in  appendix  I  of  this  report. 
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Results  in  Brief 


The  current  export  control  system  for  high  performance  computers,  which 
focuses  on  controlling  individual  machines,  is  ineffective  because  it  cannot 
prevent  countries  of  concern  from  linking  or  clustering  many  lower 
performance  uncontrolled  computers  to  collectively  perform  at  higher 
levels  than  current  export  controls  allow.  The  Department  of  Commerce 
controls  the  export  of  individual  high  performance  computers  using 
MTOPS  as  its  control  measure.  It  currently  requires  licenses  for  export  of 
computers  with  performance  levels  above  28,000  MTOPS  for  countries  of 
concern,  but  lower  performance  uncontrolled  computers  have  been 
clustered  together  from  readily  available  components  to  obtain  computing 
capabilities  up  to  70,000  MTOPS.  However,  even  if  control  thresholds  were 
raised  to  reflect  this  performance  level,  U  .S.  government  computer  experts 
and  computer  industry  representatives  agreed  that  using  MTOPS  to 
calculate  computer  performance  and  set  control  thresholds  is  an  outdated 
and  invalid  means  for  determining  whether  individual  high  performance 
computers  should  be  licensed  for  export.  They  said  that  the  measure  does 
not  account  for  new  designs  of  individual  processors  or  for  clustering 
computers  together  to  achieve  high  performance  computing  levels.  A 
Commerce  Department  technical  advisory  committee3  has  examined  three 
alternatives  to  the  current  measure  for  establishing  the  licensing  threshold, 
but  none  of  the  alternatives  would  solve  the  problems  posed  by  clustering 
uncontrolled  computer  components  to  achieve  higher  computing  levels.  In 
addition,  the  Commerce  Department  has  not  assessed  the  implications  of 
these  technological  advances  on  the  effectiveness  of  the  current  licensing 
process  to  limit  the  access  of  countries  of  concern  to  high  performance 
computing. 


3 Technical  advisory  committees,  such  as  the  committee  on  Information  Systems  Technical 
Advisory  Committee,  advise  the  Department  of  Commerce  on  the  technical  parameters  for 
export  controls  applicable  to  dual-use  commodities  and  technology  and  on  the 
administration  of  those  controls.  The  committees  comprise  representatives  from  industry 
and  government  who  represent  diverse  points  of  view  on  the  concerns  of  the  exporting 
community.  Industry  representatives  are  selected  from  firms  producing  a  broad  range  of 
goods,  technologies,  and  software  presently  controlled  for  national  security,  foreign  policy, 
nonproliferation,  and  short  supply  reasons;  or  that  are  proposed  for  such  controls,  and  are 
balanced  to  the  extent  possible  among  large  and  small  firms. 
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The  justifications  to  change  computer  export  controls  in  February  and 
August  2000,  which  the  President  provided  in  two  reports  to  the  Congress, 
are  inadequate  for  several  reasons.  First,  the  reports  address  the 
availability  of  individual  computers  to  other  countries,  but  they  do  not 
address  the  fact  that  the  performance  capabilities  of  clustered  systems  that 
can  be  acquired  by  other  countries  have  exceeded  the  new  control 
thresholds.  Second,  the  reports  address  only  selected  computer  uses  of 
military  significance  to  which  high  performance  computers  could  be 
applied  instead  of  all  such  uses  at  the  new  control  thresholds,  as  required 
by  the  1998  National  Defense  Authorization  Act.  Finally,  the  President's 
reports  make  no  reference  to  how  specific  identified  uses  at  the  new 
control  threshold  would  affect  U.S.  national  security.  Instead,  the 
President's  reports  state  that  (1)  pursuit  of  particular  national  security 
applications  would  require  more  than  computer  hardware;  (2)  there  is  little 
danger  of  most  countries  usefully  employing  high  performance  computers 
for  military  uses;  and  (3)  it  is  important  for  the  United  States  to  retain  a 
technological  advantage  in  the  design,  development,  and  production  of 
microprocessors  and  computers.  The  reports'  sections  dealing  with 
military  uses  of  high  performance  computers  and  how  such  uses  could 
effect  U  .S.  national  security  were  based  on  information  from  a  1998  study4 
used  by  the  Defense  Department  about  military  computer  uses.  The  study's 
authors  concluded  that  neither  they,  nor  the  U  .S.  government,  had 
sufficient  information  to  assess  which  countries  could  productively  use 
high  performance  computers  to  pursue  particular  applications. 

Appendix  I  discusses  the  three  alternative  measures  of  individual  computer 
performance  that  the  Department  of  Commerce's  Technical  Advisory 
Committee  has  examined.  The  appendix  also  discusses  nine  other  ideas 
that  have  been  raised  at  different  times  by  various  computer  experts  as 
possible  alternatives  to  either  the  current  MTOPS  measure  or  the  overall 
control  system.  Flowever,  neither  Commerce  nor  others  have  fully  assessed 
the  feasibility  of  these  ideas.  We  did  not  assess  the  feasibility  of 
implementing  any  of  the  ideas  but  describe  their  characteristics  and 
limitations  when  identified  by  the  experts  that  we  consulted.  As 
technological  advances  in  high  performance  computing  make  it  more 
difficult  to  maintain  the  U.S.  lead  in  military  capabilities  by  denying 


4  Seymour  E.  Goodman,  Peter  Wolcott,  and  Patrick  Homer,  High-performance  Computing, 
National  Security  Applications,  and  Export  Control  Policy  at  the  Close  of  the  20th  Century 
(Palo  Alto,  CA:  Stanford  University,  Center  for  International  Security  and  Arms  Control, 
1998). 
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advanced  technology  transfers  to  countries  of  concern,  it  may  become 
necessary  to  explore  other  options  to  maintain  the  U.S.  lead  in  military 
technology.  As  a  step  in  this  direction,  the  National  Defense  Authorization 
Act  for  fiscal  year  2000  requires  (1)  an  annual  report  on  transfers  of 
militarily  sensitive  technology  to  countries  and  entities  of  concern  and 
(2)  an  assessment  by  the  Secretary  of  Defense  on  the  cumulative  impact  of 
U.S. -granted  licenses  for  exports  of  technologies  and  technical 
information.  The  report  prepared  under  the  act  must  include  information 
on  countermeasures  that  may  be  necessary  to  overcome  the  use  of  such 
technologies  and  technical  information.  It  is  not  required,  however,  to 
assess  the  cumulative  impact  of  exports  of  nonlicensed  computers,  such  as 
those  that  can  be  clustered.  The  report  is  scheduled  for  completion  in  late 
2000. 

We  are  recommending  that  the  Secretary  of  Commerce,  in  consultation 
with  the  Secretaries  of  Defense,  E  nergy,  and  State,  convene  a  panel  of 
experts  to  conduct  a  comprehensive  assessment  and  report  to  the 
Congress  on  ways  of  addressing  the  shortcomings  of  computer  export 
controls,  including,  but  not  limited  to,  the  ideas  noted  in  this  report.  We  are 
also  recommending  that  the  Secretary  of  Defense  determine  what  U.S. 
countermeasures  are  necessary,  if  any,  to  respond  to  enhancements  of  the 
military  or  proliferation  capabilities  of  countries  of  concern  derived  from 
both  licensed  and  nonlicensed  high  performance  computing. 

Agencies  commenting  on  a  draft  of  this  report  generally  agreed  with  our 
findings  and  conclusions,  and  the  Department  of  Energy  agreed  with  our 
recommendation  to  convene  a  panel  of  experts  to  conduct  a 
comprehensive  assessment  and  report  to  the  Congress  on  ways  of 
addressing  the  shortcomings  of  computer  export  controls.  The 
Departments  of  Commerce  and  Defense  disagreed  with  the  need  to 
implement  our  recommendations  because  they  said  they  are  already 
engaged  in  interagency  reviews  of  similar  issues.  When  asked  for 
documentation  on  how  interagency  mechanisms  are  pursuing  the  points 
covered  in  our  recommendations,  the  agencies  provided  none.  As  a  result, 
we  believe  our  recommendations  are  still  valid  and  necessary  to  protect 
U  .S.  national  security  interests.  Since  Commerce  and  Defense  disagreed 
with  our  recommendations,  we  have  added  two  Matters  for  Congressional 
Consideration.  The  Congress  may  wish  to  institute  a  requirement  that  the 
Secretary  of  Commerce,  in  consultation  with  the  Secretaries  of  Defense, 

E  nergy,  and  State,  convene  a  panel  of  experts  to  conduct  a  comprehensive 
assessment  of  and  report  to  the  Congress  on  possible  ways  of  addressing 
the  shortcomings  associated  with  controlling  individual  high  performance 
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computers,  including,  but  not  limited  to,  the  ideas  noted  in  this  report.  The 
Congress  may  also  wish  to  consider  instituting  a  requirement  that  the 
Secretary  of  Defense  determine  and  report  on  what  U  .S.  countermeasures 
are  necessary,  if  any,  to  respond  to  computing-related  enhancements  of  the 
military  or  proliferation  capabilities  of  countries  of  concern,  as  we  have 
recommended  in  this  report. 


Background 


The  use  of  high  performance  computing  technology  is  reducing  the  costs 
and  time  required  for  systems  analysis,  design,  development,  test,  and 
deployment  of  military  systems  and  is  improving  the  integration  and 
effectiveness  of  complex  weapons  systems,  according  to  the  Department 
of  Defense  (DOD).  As  DOD  continues  to  reform  and  reengineer  its 
acquisition  processes,  high  performance  computing  assets  are  being  used 
to  reduce  the  number  and  cost  of  building  expensive  prototypes.  High 
performance  computing,  a  critical  enabling  technology,  is  now  a  key 
ingredient  to  the  successful  implementation  of  many  major  DOD 
acquisition  programs,  according  to  DOD.  It  is  for  these  reasons  that  the 
United  States  attempts  to  limit  the  extent  of  high  performance  computing 
capabilities  obtainable  by  countries  of  concern. 

H  igh  performance  computers  and  related  components  (for  example, 
processors)  are  controlled  under  the  Export  Administration  Act  and  the 
implementing  Export  Administration  Regulations. 5The  act  authorizes  the 
Commerce  Department  to  require  firms  to  obtain  licenses  for  the  export  of 
sensitive  items  that  may  pose  a  national  security  or  foreign  policy  concern. 
The  Departments  of  State,  E  nergy,  and  Defense  assist  Commerce,  which 
administers  the  act,  by  reviewing  export  applications  and  supporting 
Commerce  in  its  reviews  of  export  control  policy. 

Since  1993,  the  President  has  revised  U.S.  export  control  requirements  for 
high  performance  computers  five  times,  including  the  revisions  announced 
in  August  2000.  The  export  control  policy  revision  implemented  in  J  anuary 
1996  removed  license  requirements  for  most  exports  of  computers  that 
could  perform  at  or  below  a  level  of  2,000  MTOPS— an  increase  from  1,500 
MTOPS.6The  1996  policy  revision  also  organized  countries  into  four 


5  50  U.S.C.  App.  section  2401  and  following  and  15  C.F.R.  section  730  and  following. 

6  Export  Controls:  Information  on  the  Decision  to  Revise  High  Performance  Computer 
Controls  (GAO/NSI  AD-98-196,  Sept.  16, 1998). 
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computer  "tiers,"  with  each  tier  after  tier  1  representing  a  successively 
higher  level  of  concern  related  to  U  .S.  national  security  interests. 

•  Tier  1.  The  policy  placed  no  license  requirements  on  tier  1  countries, 
primarily  those  in  Western  Europe  and  J  apan. 

•  Tier  2.  Exports  of  computers  above  10,000  MTOPS  to  tier  2  countries  in 
Asia,  Africa,  Latin  America,  and  Central  and  Eastern  Europe  continued 
to  require  licenses. 

•  Tier  3.  A  dual-control  system  was  established  for  tier  3  countries. 

•  Tier  4.  Exports  of  high  performance  computers  to  tier  4  countries  were 
essentially  prohibited. 

In  1996  there  were  50  tier  3  countries,  including  China,  Russia,  India,  and 
Israel.  Since  then,  the  number  of  tier  3  countries  has  increased  to  52.  In 
J  anuary  1996,  the  threshold  for  exporting  high  performance  computers 
without  a  license  to  tier  3  countries  was  set  at  7,000  MTOPS  for  civilian  end 
users,  while  exports  of  computers  for  potential  military  end  users  at  and 
above  2,000  MTOPS  required  a  license.  Exports  of  high  performance 
computers  with  performance  capabilities  above  7,000  MTOPS  to  civilian 
end  users  in  all  tier  3  countries  required  a  license.  High  performance 
computer  exports  to  countries  in  tier  4  (for  example,  Iran,  Iraq,  and  Libya) 
were  essentially  prohibited  because  of  national  security  and  foreign  policy 
concerns  about  these  countries. 

The  Fiscal  Year  1998  National  Defense  Authorization  Act7  requires  the 
President  to  report  to  the  Congress  justifications  for  changes  in  controls 
for  computer  exports  to  certain  sensitive  countries.  The  act  further 
requires  exporters  to  notify  the  Commerce  Department  of  any  proposed 
high  performance  computer  exports  to  countries  that  pose  a  concern  for 
military  or  proliferation  reasons  to  determine  if  these  exports  need  a 
license. 8 The  act  also  provides  that  if  any  designated  agency9  raises  a 
written  objection  to  the  proposed  export  within  10  days  after  Commerce 
receives  the  notification  of  the  export,  the  export  will  require  a  license. 


7  P.L.  105-85  sec.  1211,  Nov.  1997. 

8  Advance  notification  to  Commerce  of  exporter's  intent  to  export  high  performance 
computers  will  rise  to  28,000  MTOPS,  effective  in  February  2001,  from  the  current  threshold 
of  12, 500  MTOPS. 

9  Designated  agencies  for  this  purpose  are  the  Departments  of  Defense,  E  nergy,  and  State. 
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On  J  uly  27, 1999,  February  15,  2000,  and  August  30,  2000, 10  following 
announced  changes  to  the  export  control  levels  for  high  performance 
computers,  the  President  submitted  a  report  to  the  Congress,  as  required. 
According  to  the  President's  announcements,  these  changes  were  needed 
because  of  the  extraordinarily  rapid  rate  of  technological  change  in  the 
computer  industry. 

In  the  August  2000  announcement,  the  President  described  plans  to  change 
the  controls  on  the  exports  of  high  performance  computers  by  increasing 
the  computing  performance  level  at  which  export  licenses  would  be 
required.11  These  changes  were  as  follows: 

•  Tier  1.  No  changes. 

•  Tier  2.  The  licensing  level  was  raised  to  45,000  MTOPS,  effective 
immediately. 

•  Tier  3.  The  two-level  system  for  tier  3  countries  was  eliminated,  and  the 
licensing  level  for  the  tier  was  raised  to  28,000  MTOPS,  effective 
immediately. 

•  Tier  4.  No  changes. 


Export  Control  System 
for  High  Performance 
Computers  Is  Not 
Effective 


The  current  system  of  controlling  the  export  of  individual  machines  is 
ineffective  in  limiting  countries  of  concern  from  obtaining  high 
performance  computing  capabilities  for  military  applications.  In  addition, 
information  obtained  from  the  Departments  of  Commerce,  Defense,  and 
E  nergy  and  computer  industry  representatives  shows  that,  given  advances 
in  technology,  using  MTOPS  to  establish  export  control  thresholds  is 
outdated  and  no  longer  a  valid  means  for  controlling  computing 
capabilities  and  that  an  alternative  is  needed  to  replace  it.  MTOPS  is  an 
outdated  measure  because  of  its  limited  ability  to  estimate  the  performance 
of  new  designs  of  processors  and  to  account  for  clustering  of  individual 
computers  to  achieve  high  performance  computing  capability. 


10  Summary  of  Findings  With  Respect  to  Criteria  Set  Forth  in  Subsection  1211  (d)  of  the 
National  Defense  Authorization  Act  for  Fiscal  Year  1998,  attachment  to  letter  sent  to 
Congress  (Washington,  D.C.:  The  White  House,  J  uly  27, 1999,  Feb.  15,  2000,  and  Aug.  30, 
2000). 

11  Executive  branch  agencies  involved  with  reviewing  and  recommending  changes  to 
computer  export  controls  are  the  Departments  of  Commerce,  Defense,  State,  and  E  nergy 
and  the  National  Security  Council. 
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Computer  Advances  in 
Clustered  Systems 
Compromise  the 
Effectiveness  of  Current 
Controls 


Over  the  past  several  years,  technological  advances  in  the  ability  to  cluster 
lower  performance  computers  have  resulted  in  an  increasing  capability 
worldwide  to  achieve  high  performance  computer  levels.  As  a  result, 
current  U.S.  export  controls  cannot  prevent  countries  of  concern  from 
obtaining  computing  capabilities  that  can  be  used  for  military  applications 
at  levels  much  higher  than  those  at  which  computers  are  currently 
controlled  by  the  U.S.  government. 

Commerce's  Information  Systems  Technology  Advisory  Committee,  which 
advises  and  assists  the  Secretary  of  Commerce,  reported  in  May  1999— and 
the  Lawrence  Livermore  National  Laboratory  confirmed  in  February 
2000— that  clustered  computer  systems  composed  of  readily  available 
components  have  been  built  with  a  performance  level  up  to  about  70,000 
MTOPS.  As  a  result,  the  known  performance  levels  of  clustered  systems 
exceed  even  the  latest  export  control  thresholds  set  by  the  U  .S. 
government  in  August  2000.  Figures  1  and  2  show  examples  of  clustered 
computer  systems  composed  of  readily  available  components,  such  as 
personal  computers  and  monitors,  located  at  Clemson  U  niversity  and  the 
Nehru  Centre  for  Advanced  Scientific  Research,  respectively. 
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Figure  1 :  A  Clustered  Computer  System 


Source:  Grendel  Beowulf  Workstation,  Clemson  University. 
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Source:  Jawaharlal  Nehru  Centre  for  Advanced  Scientific  Research,  India. 

According  to  a  February  2000  analysis  prepared  by  the  Lawrence 
Livermore  National  Laboratory,  "Ignoring  the  fact  that  low-performance 
computers  can  be  clustered  to  achieve  a  higher  level  of  processing  leads  to 
a  false  sense  of  security."  The  analysis  said  that  it  is  inconsistent  to  control 
the  export  of  individual  high  performance  computers  when  the 
performance  capabilities  of  clustered  computers  exceed  the  current  and 
proposed  control  levels.12  Officials  of  the  Los  Alamos  National  Laboratory 
said  that  expertise  necessary  to  build  clustered  systems  is  available 
worldwide,  including  in  countries  such  as  China  and  India,  and  that 
laboratory  officials  communicate  with  people  building  these  systems  and 
solving  the  same  types  of  technical  computing  problems  as  the  national 
laboratories.  While  discussing  the  relatively  lower  level  of  expertise 
necessary  to  put  a  clustered  computer  system  together,  the  officials 
stressed  that  it  is  more  difficult  to  operate  clustered  systems  than  to  build 


12 The  National  Security  Implications  of  Decontrolling  Export  to  Tier  III  Countries  of  High 
Performance  Computers  between  2,000  and  40,000  MTOPS.  (February  2000) 
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them.  For  example,  support  for  computer  functions  requires  having  an 
efficient  schedule  for  running  hundreds  of  separate  problems  and  the  input 
and  output  of  data  among  a  computer,  its  local  disks,  networks,  and 
archival  storage.  Without  vendor-supplied  software  to  automate  these 
functions  on  a  system  clustered  together  from  readily  available 
components,  everything  must  be  done  manually,  making  computing  a 
labor-intensive  operation. 

One  computer  facility  in  India  advertises  that  it  has  developed  a  new 
computer  system  architecture  with  clusters  of  workstations  with 
commercially  available  interconnect  technologies.  The  facility  is  on  the 
Commerce  Department's  "E  ntities  List,"  which  identifies  foreign  end  users 
that  have  been  determined  to  present  an  unacceptable  risk  of  diversion  to 
developing  weapons  of  mass  destruction  or  the  missiles  used  to  deliver 
those  weapons.  Publishing  this  list  puts  exporters  on  notice  that  any 
products  sold  to  these  end  users  may  present  concerns  and  will  require  a 
license  from  Commerce. 


MTOPS  Is  No  Longer 
Adequate  as  a  Measure  of 
Performance 


Analyses  and  information  from  U  .S.  government  officials13  and  computer 
industry  representatives  show  that  MTOPS  is  no  longer  a  valid  measure  of 
individual  computer  performance  and  needs  to  be  replaced.  During  the 
course  of  our  review,  these  officials  and  representatives  said  that  MTOPS  is 
outdated  and  can  no  longer  adequately  account  for  the  performance 
capabilities  available  from  today's  computers.  As  a  result,  the  continued 
use  of  the  MTOPS  measure  would  cause  systems  with  the  same 
performance  capabilities  to  be  treated  differently  under  export  control 
regulations,  according  to  Commerce's  Information  Systems  Technical 
Advisory  Committee. 


The  MTOPS  measure  is  unreliable  today  because  it  is  heavily,  if  not 
exclusively,  dependent  on  the  clock  rate  of  a  microprocessor,  measured  in 
megahertz,  when  several  other  factors  should  be  considered.  It  does  not 
account  for  certain  factors,  such  as  memory  retrieval  times, 
interconnection  methods,  and  internal  bus  speeds.  (A  bus  is  a  collection  of 
wires  through  which  data  is  transmitted  from  one  part  of  a  computer  to 
another.)  An  internal  bus  connects  all  the  internal  computer  components  to 
the  central  processing  unit  and  main  memory.  The  size  of  a  bus  is  important 


13  U.S.  government  officials  include  the  Deputy  Under  Secretary  of  Defense  (Science  and 
Technology)  and  officials  of  the  Commerce  and  E  nergy  Departments. 
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because  it  determines  how  much  data  can  be  transmitted  at  one  time.  A 
fast  bus  allows  data  to  be  transferred  faster,  which  makes  applications  run 
faster.  In  networking,  a  bus  is  a  central  cable  that  connects  all  devices  on  a 
local  area  network.  In  addition,  MTOPS  is  inaccurate  if  incorrectly  applied 
to  computing  elements  organized  in  certain  ways,  such  as  computer 
clusters.  U  nder  DOD  auspices,  U  .S.  government  officials,  computer 
industry  representatives,  and  academics  met  in  December  1997  to 
determine  whether  the  use  of  MTOPS  is  sufficient  to  rate  the  relative 
performance  of  current  and  future  computer  systems.  The  group 
determined  that  there  were  variances  of  a  factor  of  2  in  the  performance  of 
delivered  systems  relative  to  the  MTOPS-calculated  performance  because 
of  the  wide  range  of  architectures  in  use,  but  it  concluded  that  continuing 
rapid  changes  in  technology  might  result  in  yet  larger  variances  in  the  near 
future. 


President's 
J  ustification  for 
Changing  Computer 
Export  Control 
Thresholds  Was 
Inadequate 


The  National  Defense  Authorization  Act  of  1998  requires  the  President  to 
provide  a  report  to  the  Congress  on  three  factors  when  proposing  a 
modification  of  the  export  control  thresholds  for  high  performance 
computers.  First,  the  report  is  to  address  the  availability  of  high 
performance  computers  from  other  countries.  Second,  it  is  to  address  all 
potential  computer  uses  of  military  significance  at  the  new  control 
thresholds.  Third,  the  report  is  to  assess  the  impact  of  such  uses  on  U.S. 
national  security  interests.  In  response  to  the  act's  requirements,  the 
February  and  August  2000  reports  presented  information  on  the  availability 
of  individual  computers  from  other  countries  but  did  not  recognize  such 
countries'  capabilities  to  cluster  computers  to  obtain  computing 
performance  beyond  the  new  thresholds.  Furthermore,  the  August  report 
addressed  only  22  of  172  known  military  uses  of  high  performance 
computers  up  to  the  new  control  threshold.  Finally,  the  report  made  no 
reference  to  the  national  security  impact  of  specific  identified  uses  at  the 
new  control  thresholds.  Instead,  the  report  stated  that  (1)  pursuit  of 
particular  national  security  applications  would  require  more  than  computer 
hardware;  (2)  there  was  little  danger  of  most  tier  2  and  3  countries 
diverting  high  performance  computers  from  other  purposes  and  usefully 
employing  them  for  military  uses;  and  (3)  it  was  important  for  the  United 
States  to  retain  a  technological  advantage  in  the  design,  development,  and 
production  of  microprocessors  and  computers.  The  reports  relied  for  their 
information  about  national  security  impacts  of  military  computer  uses  on  a 
study  that  stated  that  it  could  not  assess  which  countries  of  concern  could 
use  computers  for  particular  military  applications  because  the  U  .S. 
government  did  not  have  information  to  make  such  an  assessment. 
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Clustering  Not  Factored 
Into  New  Thresholds  in 
Considering  Computing 
Performance  Availability  in 
Other  Countries 


The  President's  reports  were  required  to  address  the  extent  to  which  high 
performance  computers  with  capabilities  between  the  established  level  and 
the  new  proposed  level  of  performance  are  available  from  other  countries. 
In  response  to  the  requirement,  the  reports  to  the  Congress  justified  the 
new  control  thresholds  based  on  industry-provided  information  indicating 
the  MTOPS  performance  of  mass  market  processors  that  the  industry 
expected  would  be  produced  within  the  next  6  months.  The  reports  stated 
that  such  processors  would  be  uncontrollable  worldwide.  While  the 
administration  discussed  computer  clustering  as  contributing  to  such 
availability,  it  did  not  factor  computer  clustering  into  its  control  threshold 
changes  made  during  2000,  nor  did  it  indicate  that  with  clustered  systems 
other  countries  could  obtain  performance  capabilities  that  exceed  the  new 
control  thresholds.  If  they  had  presented  this  fact  in  the  reports,  it  would 
have  revealed  that  the  computer  control  system  was  ineffective. 


Although  the  President's  February  and  August  2000  reports  discussed 
advances  in  clustering,  they  did  not  indicate  that  performance  capabilities 
of  clustered  systems  have  exceeded  the  new  control  thresholds.  The 
President's  reports  noted  that  advanced  network  technologies,  particularly 
in  parallel  processing  known  as  "distributed  computing,"  are  freely 
available.  Cost  considerations  and  advances  in  computer  technology  in 
both  the  United  States  and  overseas  have  created  a  favorable  environment 
for  high  performance  distributed  computing  using  readily  available 
hardware  and  software,  according  to  the  reports.  The  reports  added  that 
relatively  low-cost  high  performance  computer  systems  typically  consist  of 
large  clusters  of  commercially  available  workstations  or  personal 
computers  that  are  linked  by  interconnection  hardware  and  high-speed 
communication  software.  As  a  result,  the  affordability  and  widespread 
availability  of  these  computer  products  permit  foreign  end  users  to 
configure  these  commercially  available  products  into  high  performance 
computer  systems.  The  President's  reports  did  not  state  that  high 
performance  computing  up  to  about  70,000  MTOPS  is  attainable. 

While  computer  clustering  was  not  factored  into  the  control  threshold 
changes  made  during  2000,  the  President's  August  2000  report  justified  the 
new  control  thresholds  based  on  industry  information  indicating  the 
MTOPS  performance  of  mass  market  processors  to  be  produced  within  the 
next  6  months  that  Commerce  officials  believed  would  be  uncontrollable. 
As  of  August  2000,  the  fastest  mass-marketed  microprocessor  had  a  peak 
performance  measure  between  3,000  and  4,000  MTOPS.  By  the  fall  2000,  a 
new  6,100  MTOPS  microprocessor  was  expected  to  be  available,  and  a 
four-microprocessor  computer  based  on  this  new  microprocessor  was 
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expected  to  have  a  performance  level  of  about  24,000  MTOPS.  The  August 
report  further  stated  that  the  next  generation  of  computers  with  eight 
microprocessors  and  a  performance  of  26,000  MTOPS,  is  expected  to  be 
widely  available  in  early  2001.  Consequently,  for  end  users  in  tier  3 
countries,  thresholds  were  set  at  28,000  MTOPS,  slightly  higher  than  the 
expected  availability  of  the  26,000  MTOPS  machine.  The  report  stated  that 
computer  manufacturers  in  the  United  States  and  in  foreign  countries 
continue  to  produce  computers  that  are  smaller,  cheaper,  and  easier  to 
install  and  maintain  but  are  more  powerful  than  ever  before.  This  trend  is 
due  in  large  part  to  rapid  advances  in  microprocessor  technology, 
according  to  the  report.  Computer  companies  projected  that  their  chips 
and  systems  would  be  produced  in  the  tens  of  thousands  per  month, 
depending  on  the  specific  processor. 


All  Potential  Computer  Uses  The  National  Defense  Authorization  Act  of  1998  requires  that  the 
Not  Addressed  President's  report  to  the  Congress  address  all  potential  uses  of  military 

significance  to  which  high  performance  computers  at  the  new  levels  could 
be  applied.  However,  the  President's  reports  to  the  Congress  during  2000 
did  not  address  all  such  computer  applications,  even  though  this 
information  was  available  from  the  1998  Department  of  Defense-  and 
Commerce-sponsored  study  that  was  used  as  the  basis  for  these  sections  of 
the  reports.  DOD  officials  from  the  Defense  Threat  Reduction  Agency, 
which  is  responsible  for  sections  of  the  reports  on  military  uses  of  high 
performance  computers,  told  us  that  they  took  their  information  from  the 
1998  study  because  the  agency  did  not  have  the  resources  to  initiate  an 
independent  review  of  this  and  related  issues.  The  officials  chose  to  report 
on  3  of  10  categories  of  applications  identified  by  the  1998  study. 
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We  identified  a  total  of  22  computer  applications  mentioned  in  the 
President's  report:  17  applications  that  can  be  operated  up  to  a 
performance  level  of  21,000  MTOPS  and  5  applications  that  can  be 
operated  at  nonspecified  levels  between  1,000  and  115,000  MTOPS. 14The 
1998  Defense-  and  Commerce-sponsored  study,  however,  specifically 
identified  172  military  applications  that  were  run  on  computers  up  to 
28,000  MTOPS.15  Of  these  military  applications,  47  were  run  on  computers 
between  20,000  and  28,000  MTOPS  (the  previous  tier  3  computer  control 
thresholds  for  civilian  end  users  and  newly  announced  tier  3  computer 
control  thresholds  for  all  end  users).  Appendix  IV  lists  the  applications  of 
national  security  importance  identified  for  the  1998  Defense-  and 
Commerce-sponsored  study. 


Information  Provided  on 
National  Security  Impacts 
Was  Inadequate 


The  National  Defense  Authorization  Act  of  1998  requires  that  the 
President's  report  to  the  Congress  assess  the  impact  on  U.S.  national 
security  interests  of  potential  uses  of  military  significance  to  which  high 
performance  computers  at  the  new  levels  could  be  applied.  The  President's 
reports,  however,  did  not  specifically  discuss  the  impacts  on  U.S.  national 
security  interests  for  any  of  the  national  security  applications  that  the 
report  identified,  as  required  by  the  1998  National  Defense  Authorization 
Act.  The  reports  made  general  statements  that  most  countries  cannot 
effectively  use  high  performance  computers  and  that  there  are  therefore  no 
national  security  impacts  on  the  United  States.  This  1998  study,  however, 
stated  that  the  U  .S.  government  provided  insufficient  information  to  assess 
this  issue. 


Although  the  President's  August  report  identified  only  22  of  the  172  military 
applications  that  were  run  on  computers  up  to  28,000  MTOPS,  the  report 
did  not  discuss  the  national  security  impacts  on  the  United  States  of 
Russia,  China,  or  other  countries  obtaining  high  performance  computing  up 


14The  President's  reports  do  not  clearly  distinguish  computer  applications  by  name  and 
performance  levels.  As  a  result,  it  is  impossible  in  all  cases  to  determine  from  the  reports 
whether  mentioned  applications  are  distinct  or  identical. 

15  According  to  the  1998  Department  of  Defense-  and  Commerce-sponsored  study,  the 
absence  of  a  particular  type  of  problem  at  some  performance  level  should  not  be  interpreted 
as  a  statement  that  no  version  of  that  application  can  be  solved  at  that  performance  level. 
MTOPS  are  only  rough  indicators  of  the  performance  level  required  for  a  particular  kind  of 
application,  That  a  given  problem  was  run  on  a  machine  with  a  particular  MTOPS  level  does 
not  mean  that  all  systems  with  a  greater  MTOPS  level  can  solve  the  problem  or  that  all 
systems  with  a  lower  MTOPS  level  cannot. 
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to  the  new  control  thresholds,  even  for  those  applications  identified.  The 
President's  reports  also  did  not  identify  which  countries  of  concern  could 
gain  the  hardware  capability  to  conduct  militarily  useful  applications  at  the 
new  control  thresholds.  For  example,  the  reports  stated  that  Russia  and 
China  have  demonstrated  that  they  have  the  expertise  necessary  to  use 
high  performance  computers  for  particular  national  security  applications, 
such  as  developing  submarines,  advanced  aircraft,  composite  materials,  or 
a  variety  of  other  devices.  The  reports  also  did  not  discuss  other  countries 
that  can  use  high  performance  computing  for  military  applications.  India, 
for  example,  which  has  organizations  that  we  identified  as  both  using  high 
performance  computing  for  military  purposes  and  being  listed  on  the 
Commerce  Department's  "E  ntities  List"  of  end  users  of  proliferation 
concern,  was  not  mentioned  in  the  President's  reports. 

The  reports  stated  that  most  tier  2  and  3  countries  have  little  or  no 
experience  in  a  host  of  national  security  applications  and  there  is  little 
danger  of  these  countries  diverting  high  performance  computers  from 
other  pressing  needs  and  usefully  employing  them  to  develop  military 
items.  Accordingto  DOD  officials  responsible  for  the  sections  of  the  report 
on  military  uses  of  high  performance  computers,  the  1998  Defense-  and 
Commerce-sponsored  study  was  the  basis  for  these  statements.  The  1998 
study,  however,  indicated  that  it  did  not  have  sufficient  information  to 
assess  the  impact  of  militarily  significant  applications  on  U.S.  national 
security  interests.  The  study  stated  that 

"A  critical  question,  which  we  have  been  unable  to  pursue  satisfactorily  in  this  study,  is 
which  countries  are  able  to  productively  use  [high  performance  computing]  to  pursue  which 
applications?  We  have  requested  such  information  from  the  U .S.  national  security 
community,  but  have  received  few  answers.  It  does  not  appear  that  the  U.S.  government  is 
effectively  gathering  such  intelligence  in  a  systematic  fashion.  More  specifically,  the  U  ,S. 
government  does  not  appear  to  have  as  good  an  understanding  of  individual  end  use 
organizations  of  concern  as  is  needed  by  the  export  control  regime." 

The  President's  reports  did  not  state  which  countries  can  use  high 
performance  computers  for  particular  military  applications  or  which 
countries  cannot,  and  the  analysis  that  was  absent  in  the  1998  Defense-  and 
Commerce-sponsored  study  and  an  earlier  1995  study  has  not  been  done. 
As  we  reported  in  1998, 16  the  principal  author  of  the  Defense-  and 
Commerce-sponsored  study  and  DOD  officials  told  us  that  no  assessment 
had  been  done  in  1995  to  determine  how  national  security  would  be 


16  Export  Contro/s  (GAO/NSIAD-98-196,  Sept.  16, 1998). 
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impacted  if  high  performance  computers  were  exported  to  particular 
countries  of  concern  and  what  military  advantages  such  countries  could 
achieve.  The  1998  study  reiterated  this  position.  We  recommended  in  1998 
that  the  Secretary  of  Defense  do  such  an  evaluation,  specifically  addressing 
(1)  how  and  at  what  performance  levels  countries  of  concern  use  high 
performance  computers  for  military  modernization  and  proliferation 
activities,  (2)  the  threat  of  such  uses  to  U  .S.  national  security  interests,  and 
(3)  the  extent  to  which  such  high  performance  computers  are  controllable. 
This  recommendation  has  not  been  implemented.  Although  DOD  stated 
that  the  interagency  process  had  considered  these  factors  in  the  1995 
review  of  computer  export  controls  and  would  consider  them  in  any  future 
review,  it  provided  no  evidence  that  this  has  occurred. 

An  annual  report  on  transfers  of  militarily  sensitive  technology  to  countries 
and  entities  of  concern,  required  under  section  1402  of  the  National 
Defense  Authorization  Act  for  fiscal  year  2000,  might  partially  address  the 
question  of  the  national  security  impact  of  high  performance  computer 
uses  of  military  significance.  It  is  to  include  an  assessment  by  the  Secretary 
of  Defense,  in  consultation  with  thej  oint  Chiefs  of  Staff  and  the  Director  of 
Central  Intelligence,  on  the  cumulative  impact  of  U.S. -granted  licenses  for 
exports  of  technologies  and  technical  information  to  these  countries  and 
entities.  Thus,  the  study  would  be  expected  to  include  licensed  high 
performance  computers  as  part  of  the  assessment.  This  report  is  also 
required  to  include  information  on  countermeasures  that  may  be  necessary 
to  overcome  the  use  of  such  technologies  and  technical  information. 
However,  it  is  not  required  to  include  an  assessment  of  the  cumulative 
impact  of  nonlicensed  computer  exports,  such  as  those  that  could  be 
clustered.  The  report  is  scheduled  for  completion  by  the  end  of  2000. 


Conclusions  The  currer|t  export  control  system  for  computers,  as  implemented  by  the 

Department  of  Commerce  and  which  focuses  on  controlling  the  export  of 
individual  computers,  is  not  effective  in  limiting  countries  of  concern  from 
obtaining  high  performance  computing  capabilities.  As  a  result  of  advances 
in  clustering  technology,  countries  of  concern  can  obtain  computing 
capabilities  above  current  U  .S.  export  control  levels  necessary  for  military 
applications.  It  is  also  now  widely  recognized  within  the  Departments  of 
Commerce,  Defense,  and  E  nergy  and  the  computer  industry  that  the  use  of 
MTOPS  as  the  means  of  determining  export  control  thresholds  is  outdated, 
no  longer  a  valid  means  for  controlling  computing  capabilities,  and  needs 
to  be  replaced.  In  light  of  these  now  well  established  developments,  the 
process  needs  to  be  reexamined,  and  potential  alternative  ways  to 
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safeguard  U.S.  national  security  interests  related  to  high  performance 
computer  exports  need  to  be  explored. 

In  addition,  the  President's  reports  justifying  computer  control  changes 
continue  to  be  inadequate.  It  is  a  particularly  important  omission  for  the 
U.S.  government  to  not  assess  the  national  security  impacts  of  how 
countries  of  concern  can  use  high  performance  computers  at  successively 
higher  performance  levels  for  military  purposes.  Although  we  have 
highlighted  this  issue  in  the  past,  the  administration  has  presented  no 
further  assessment.  As  advances  in  high  performance  computing  make  it 
more  difficult  to  maintain  the  U.S.  technological  lead  in  military 
capabilities  by  denying  advanced  technology  transfers  to  countries  of 
concern,  it  may  become  necessary  to  explore  other  options  to  maintain  the 
U.S.  military  technology  lead.  For  example,  the  National  Defense 
Authorization  Act  for  fiscal  year  2000  required  the  Secretary  of  Defense  to 
assess  the  cumulative  impact  of  U.S.  export  licenses— including  high 
performance  computing  licenses— and  possible  countermeasures  that  may 
be  necessary  to  overcome  the  use  of  such  technologies  and  technical 
information.  However,  it  did  not  require  an  assessment  of  the  cumulative 
impact  of  exporting  nonlicensed  computers  that  could  be  clustered,  or  of 
potential  countermeasures  to  such  an  impact. 


Recommendations  for 
Executive  Action 


Since  the  current  export  control  system  for  high  performance  computers 
cannot  prevent  countries  of  concern  from  obtaining  high  performance 
computing  capabilities,  we  recommend  that  the  Secretary  of  Commerce,  in 
consultation  with  the  Secretaries  of  Defense,  Energy,  and  State,  convene  a 
panel  of  experts  to  comprehensively  assess  and  report  to  the  Congress  on 
possible  ways  of  addressing  the  shortcomings  of  computer  export  controls, 
including,  but  not  limited  to  the  ideas  noted  in  this  report.  For  example,  the 
Commerce  Department's  National  Institute  of  Standards  and  Technology, 
which  researches  computer  systems'  performance  and  promotes  the 
effective  evaluation  and  efficient  use  of  advanced  computers,  might 
participate  in  the  panel  because  it  designs  evaluations  that  economically 
and  reliably  characterize  high  performance  computer  designs.  This 
assessment  should  report  on  the  costs  and  benefits  of  each  proposed  idea, 
including  its  technical  feasibility. 


In  addition,  the  report  required  by  the  National  Defense  Authorization  Act 
of  2000  concerning  countermeasures  that  may  be  necessary  to  overcome 
the  use  of  sensitive  technologies  and  technical  information  exported  to 
countries  of  concern  is  not  required  to  include  an  assessment  of  the 
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cumulative  impact  of  exports  of  nonlicensed  computers,  such  as  those  that 
could  be  clustered.  Therefore,  we  recommend  that  the  Secretary  of 
Defense  determine  what  U.S.  countermeasures  are  necessary,  if  any,  to 
respond  to  computing-related  enhancements  of  the  military  or  proliferation 
capabilities  of  countries  of  concern. 


Agency  Comments  and 
Our  Evaluation 


The  Department  of  Commerce  provided  written  comments  (see  app.  IV)  on 
a  draft  of  this  report.  The  Department  of  E  nergy's  Acting  Chief  Operating 
Officer  for  the  National  Nuclear  Security  Administration  and  the  Principal 
Deputy  Under  Secretary  of  Defense  for  Policy  provided  oral  comments. 
The  Department  of  Energy  concurred  with  the  report's  findings  and 
conclusions  and  our  recommendation  concerning  the  convening  of  a  panel 
of  experts  and  said  that  the  Department  looks  forward  to  participating  with 
the  Secretaries  of  Commerce,  Defense,  and  State  in  such  a  panel  to  conduct 
a  comprehensive  assessment  of  possible  ways  of  addressing  the 
shortcomings  of  computer  export  controls.  The  Departments  of  Commerce 
and  Defense  generally  agreed  with  the  report's  findings  and  conclusions 
relating  to  the  problems  we  identified  with  clustering  technology  and  the 
continuing  use  of  MTOPS  as  a  measure  for  controlling  individual  high 
performance  computers.  However,  they  disagreed  with  the  need  to 
convene  an  expert  panel  or  to  determine  what  countermeasures  are 
necessary  to  respond  to  enhancements  of  military  capabilities  of  countries 
of  concern  which  such  countries  may  have  gained  through  the  use  of  high 
performance  computing. 


The  Departments  of  Commerce  and  Defense  said  that  they  disagreed  with 
the  need  to  implement  our  recommendation  that  the  Secretary  of 
Commerce  and  other  secretaries  convene  a  panel  of  experts  to 
comprehensively  assess  possible  ways  of  addressing  the  shortcomings  of 
computer  export  controls  because  they  said  the  administration  and  experts 
are  already  studying  similar  issues.  Commerce  stated  that  it  would  be 
counterproductive  and  unnecessary  to  convene  a  panel  of  experts,  since 
the  administration  was  already  examining  this  issue  through  existing 
interagency  mechanisms  involving  DOD,  E  nergy,  and  State.  Commerce  also 
said  that  the  Center  for  Strategic  and  International  Studies  has  a  broad 
group  of  experts  currently  studying  the  same  issues  as  encompassed  in  the 
recommendation.  DOD  said  that  the  administration  has  been  consulting 
with  experts  to  formulate  an  approach  for  controlling  computing 
capabilities  and  is  assessing  various  alternatives—  including  "some" 
identified  in  our  draft  report.  Subsequent  to  receiving  these  comments,  we 
asked  both  Departments  to  describe  in  more  detail  and  document  what 
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comprehensive  study  was  being  conducted.  Neither  Department  provided 
any  additional  information  to  show  that  there  was  an  interagency  study 
being  conducted  as  we  are  recommending.  Although  we  sought 
information  on  steps  being  taken  that  would  address  the  issues  covered  by 
our  recommendations,  neither  the  Commerce  Department  nor  Department 
of  Defense  provided  any  evidence  that  an  interagency  mechanism  is 
systematically  examining  alternative  ways  to  protect  U  .S.  national  security 
interests  related  to  the  shortcomings  of  computer  export  controls—  beyond 
the  narrowly  focused  effort  to  seek  alternatives  to  the  MTOPS  measure  for 
controlling  individual  machines. 

Regarding  the  study  by  the  Center  for  Strategic  and  International  Studies, 
we  note  that  it  is  tasked  to  "develop  the  framework  for  a  new  effective . . . 
agreement  that  would  regulate  certain  militarily  useful  goods  and 
technologies  on  a  multilateral  basis,"  not  to  examine  the  costs,  benefits, 
and  technical  feasibility  of  each  possible  way  identified  in  this  report  of 
addressing  the  shortcomings  associated  with  controlling  individual  high 
performance  computers. 

The  Departments  of  Commerce  and  Defense  also  disagreed  with  the  need 
to  implement  our  recommendation  that  the  Secretary  of  Defense  determine 
what  U.S.  countermeasures  are  necessary  to  respond  to  enhancements  of 
the  military  capabilities  of  countries  of  concern  from  both  licensed  and 
nonlicensed  high  performance  computing.  Commerce  stated  that  the  best 
countermeasure  is  to  ensure  that  the  U  .S.  military  and  defense  industries 
continue  to  have  access  to  the  computer  technology  needed  to  maintain  the 
U.S.  military  advantage.  DOD  stated  that  the  Department  continuously 
assesses  the  military  capabilities  of  potential  adversaries  and  countries  of 
concern  to  identify  threats  to  U  .S.  forces  and  described  the  assessment 
process.  While  we  agree  that  maintaining  the  U.S.  technological  advantage 
in  computing  is  important,  we  note  that  neither  Commerce  nor  Defense 
specified  how  or  whether  DOD  (1)  has  assessed  and  identified  any  threats 
posed  by  high  performance  computing  to  U  .S.  national  security  interests  or 
(2)  identified  and  implemented  countermeasures  to  such  threats. 

F urthermore,  ongoing  studies  tasked  to  assess  the  need  for 
countermeasures  related  to  U  .S.  exports  of  sensitive  technologies  are  not 
reviewing  threats  posed  by  nonlicensed  clustered  computer  systems,  a 
factor  that  we  believe  is  critical  to  assess. 

DOD  disagreed  with  our  conclusion  that  the  President's  February  and 
August  2000  reports  to  the  Congress  were  inadequate  with  regard  to 
required  national  security  assessments  accompanying  changes  to  computer 


Page  23 


GAO-Ol-lO  Export  Controls 


control  levels.  DOD  stated  that  the  President's  reports  have  consistently 
noted  that  most  activities  associated  with  various  aspects  of  military 
capabilities  benefit  from  some  computing  capabilities.  It  further  stated  that 
the  President's  reports  to  the  Congress  have  provided  examples  of  national 
security  applications  to  illustrate  that  the  range  of  military  applications  that 

(1)  benefit  from  high  performance  computing  is  "almost  ubiquitous"  and 

(2)  can  be  performed  on  almost  any  computer  is  extensive.  Nonetheless,  as 
this  report  and  our  prior  reports  have  pointed  out,  we  continue  to  believe 
that  the  President's  reports  did  not  adequately  provide  the  information  that 
the  law  requires.  The  President's  reports  did  not  address  all  potential  uses 
of  military  significance  to  which  high  performance  computers  at  the  new 
levels  could  be  applied— even  though  the  administration  possessed 
information  on  such  uses  in  the  1998  Defense-  and  Commerce-sponsored 
study  (part  of  which  we  have  reprinted  in  app.  Ill)— and  did  not  assess  the 
impact  of  such  uses  on  U  .S.  national  security  interests. 


Matters  for 

Congressional 

Consideration 


Since  the  Departments  of  Commerce  and  Defense  disagreed  with 
recommendations  which  we  believe  are  still  valid  and  needed,  the 
Congress  may  wish  to  institute  a  requirement  that  the  Secretary  of 
Commerce,  in  consultation  with  the  Secretaries  of  Defense,  E  nergy,  and 
State,  convene  a  panel  of  experts  to  conduct  a  comprehensive  assessment 
of  and  report  to  the  Congress  on  possible  ways  of  addressing  the 
shortcomings  associated  with  controlling  individual  high  performance 
computers,  including,  but  not  limited  to,  the  ideas  noted  in  this  report. 

In  addition,  to  address  the  issue  of  countermeasures  that  may  be  necessary 
to  overcome  the  use  of  sensitive  technologies  and  technical  information 
exported  to  countries  of  concern,  the  Congress  may  also  wish  to  consider 
instituting  a  requirement  that  the  Secretary  of  Defense  determine  and 
report  on  what  U.S.  countermeasures  are  necessary,  if  any,  to  respond  to 
computing-related  enhancements  of  the  military  or  proliferation 
capabilities  of  countries  of  concern,  as  we  have  recommended  in  this 
report. 


Scope  and 
Methodology 


To  assess  the  effectiveness  of  current  export  controls  in  preventing 
countries  of  concern  from  obtaining  high  performance  computing 
capabilities  that  can  be  used  for  military  applications,  we  reviewed  studies 
and  briefing  slides  by  U.S.  government  agencies  and  computer  industry 
technical  specialists  that  described  technological  computing  advances  and 
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assessed  the  MTOPS  measure.  Such  studies  included  assessments  by  the 
Lawrence  Livermore  and  Sandia  National  Laboratories  on  possible 
alternatives  to  current  computer  controls.  We  interviewed  officials  from 
the  Departments  of  Defense,  Commerce,  State,  and  E  nergy;  the  Central 
Intelligence  Agency;  and  the  three  major  national  weapons  laboratories, 
Lawrence  Livermore,  Los  Alamos,  and  Sandia.  We  interviewed  officials 
from  major  computer  manufacturers,  Compaq,  Hewlett  Packard,  IBM,  and 
SUN  Microsystems,  as  well  as  the  computer  scientist  responsible  for  the 
Top  500  List  of  the  most  advanced  high  performance  computers  in  the 
world.  In  addition,  we  observed  meetings  of  Commerce  Department's 
Information  Systems  Technology  Advisory  Committee. 

To  evaluate  the  President's  justifications  for  changing  the  computer  control 
thresholds,  we  reviewed  the  President's  February  2000  and  August  2000 
reports  to  the  Congress  justifying  changes  to  computer  control  thresholds 
and  a  White  House  fact  sheet  detailing  the  changes  made  in  August  2000. 
We  reviewed  Commerce  Department  analyses  and  computer  industry 
information  regarding  projected  production  schedules  and  technical 
performance  ratings  for  processors  scheduled  to  be  marketed  in  the  next 
6  to  12  months.  In  addition,  we  reviewed  studies  and  briefing  slides  by  U.S. 
government  agencies  and  computer  industry  technical  specialists  that 
described  technological  computing  advances  and  assessed  the  MTOPS 
measure.  We  interviewed  officials  from  the  Departments  of  Defense, 
Commerce,  State,  and  Energy;  the  Central  Intelligence  Agency;  and  the 
three  major  national  weapons  laboratories,  Lawrence  Livermore,  Los 
Alamos,  and  Sandia.  We  interviewed  officials  from  major  computer 
manufacturers,  Compaq,  Hewlett  Packard,  IBM,  and  SUN  Microsystems,  as 
well  as  the  computer  scientist  responsible  for  the  Top  500  List  of  the  most 
advanced  high  performance  computers  in  the  world.  In  addition,  we 
observed  meetings  of  the  Commerce  Department's  Information  Systems 
Technology  Advisory  Committee. 

We  conducted  our  review  from  February  through  November  2000  in 
accordance  with  generally  accepted  government  auditing  standards. 


Unless  you  publicly  announce  the  contents  of  this  report  earlier,  we  plan  no 
further  distribution  of  this  report  until  15  days  from  its  issue  date.  At  that 
time,  we  will  send  copies  of  this  report  to  other  appropriate  congressional 
committees;  the  Honorable  William  S.  Cohen,  Secretary  of  Defense;  the 
Honorable  Madeleine  K.  Albright,  Secretary  of  State;  the  Honorable 
Norman  T.  M  ineta,  Secretary  of  Commerce;  and  the  Honorable  William 
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Richardson,  Secretary  of  Energy.  Copies  will  also  be  made  available  to 
others  upon  request. 

If  you  or  your  staff  have  any  questions  concerning  this  report,  please  call 
me  at  (202)  512-4128  or  Mr.  Rhodes  at  (202)  512-6412.  Key  contributors  to 
this  assignment  were  Claude  Adrien,  J  effrey  D.  Phillips,  F.  J  ames  Shafer, 
and  Hai  Tran. 


Harold  J .  J  ohnson 
Director,  International  Affairs 
and  Trade 


Keith  A.  Rhodes 
Chief  Technologist 
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Appendix  I 

Possible  Ways  of  Safeguarding  U.S.  National 
Security  Interests  Related  to  Computer 
Exports 


To  identify  possible  alternative  ways  to  safeguard  U.S.  national  security 
interests  related  to  computer  exports,  we  reviewed  analyses  and  other 
documentation  prepared  by  the  two  Department  of  E  nergy  national 
laboratories  addressing  this  question,  Lawrence  Livermore  and  Sandia.  We 
also  interviewed  officials  from  the  Departments  of  Defense,  Commerce, 
State,  and  Energy— including  LosAlamos,  Lawrence  Livermore,  and  Sandia 
National  Laboratories;  the  Central  Intelligence  Agency;  and  officials  from 
major  computer  manufacturers,  Compaq,  Hewlett  Packard,  IBM,  and  SUN 
Microsystems. 

We  identified  12  ideas  for  addressing  shortcomings  to  the  current  export 
control  system.  These  ideas  have  not  been  comprehensively  evaluated  by 
the  U  .S.  government.  The  Department  of  Commerce  has  considered  the 
first  three  ideas  we  describe  in  this  appendix,  but  none  of  the  three 
addresses  the  export  control  problem  created  by  clustering  technology.  We 
also  present  nine  additional  ideas  identified  through  our  research  and 
discussions  with  experts.  We  did  not  assess  the  feasibility  of  implementing 
any  of  the  ideas  but  rather  describe  their  characteristics  and  limitations 
when  identified  by  the  experts.  It  is  important  to  note  that  these  ideas  have 
not  been  assessed  for  their  feasibility  to  replace  the  current  export  control 
mechanism;  moreover,  most  of  them  do  not  address  the  challenge  created 
by  advances  in  clustering  technology. 


Alternatives  to  Using 
MTOPS  Considered  by 
the  Commerce- 
Sponsored  Technical 
Advisory  Committee 


The  three  ideas  that  the  Commerce-sponsored  I  nformation  Systems 
Technical  Advisory  Committee  (ISTAC)  reviewed  are  (1)  counting 
processors,  (2)  measuring  power  dissipation,  and  (3)  indexing  control 
thresholds  to  a  common  benchmark. 


Counting  Processors  Using  this  approach,  computers  would  be  controlled  by  counting  the 

I  n Stead  of  U  sing  MTOPS  number  of  processors  in  each  computing  system.  For  example,  a  license 

might  be  required  if  a  computer  to  be  exported  to  tier  3  countries  contained 
eight  or  more  processors,  while  computers  with  fewer  processors  could  be 
exported  without  a  license.  Presumably,  the  precise  number  of  processors 
for  each  tier  would  be  determined  as  the  details  of  such  a  proposal  were 
finalized. 
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Appendix  I 

Possible  Ways  of  Safeguarding  U.S.  National 
Security  Interests  Related  to  Computer 
Exports 


Computer  industry  representatives  and  I  STAC  members  stated  that 
counting  processors  would  be  simpler  than  using  the  MTOPS  measure. 
Under  current  controls,  computer  chip  manufacturers  must  calculate  the 
MTOPS  rating  for  each  processor  they  manufacture,  and  computer 
companies  must  calculate  the  MTOPS  rating  for  each  of  the  computer 
systems  they  produce.  Department  of  Energy  and  ISTAC  officials  also 
noted  that  by  counting  processors,  they  could  better  adapt  to  changes  in 
computer  technology  because  advances  in  processors  used  in  a  system 
would  not  result  in  the  need  to  change  the  export  controls.  On  the  other 
hand,  ISTAC  members  said  that  counting  processors  is  no  more  accurate 
an  indicator  than  MTOPS  for  indicating  a  level  of  computer  performance. 
Furthermore,  despite  the  intended  simplicity  of  the  approach,  the 
Committee  members  noted  that  there  is  no  consensus  within  the  computer 
community  on  the  definition  of  a  processor.  They  also  said  that  exceptions 
to  the  counting  procedure  would  need  to  be  determined  for  high 
performance  single  processors  and  for  processors  and  computers  with 
unique  designs.  For  example,  some  processors  today  include  advanced 
graphics  or  memory  capabilities,  but  others  do  not  and  therefore  have 
different  performance  capabilities. 


Measuring  Power 
Dissipation 


Power  dissipation  is  a  measure  of  watts  per  MTOPS  that  is  currently  used 
in  J  apan  to  determine  the  environmental  effects  of  computers.  The 
measure  produces  a  ratio  of  the  watts  per  MTOPS  to  the  size  of  the  box 
housing  the  microprocessor.  The  May  2000  ISTAC  paper  stated  that  this 
alternative  was  briefly  studied  but  rejected  because  of  its  inability  to 
accurately  measure  the  performance  capabilities  of  a  system.  Two  systems 
with  the  same  performance  capability  might  have  different  ratios  because 
each  is  housed  in  a  different  sized  box  with  different  power  supplies  and 
different  disk  drives,  according  to  the  Committee  paper.  As  a  result,  the 
system  measured  to  have  a  lower  ratio  would  be  controlled  less  stringently 
than  the  smaller  system. 


Indexing  Control 
Thresholds  to  a  Common 
Benchmark 


Computer  export  control  levels  could  be  set  by  using  a  common 
performance  benchmark.  The  May  2000  ISTAC  paper  on  alternative  control 
measures1  also  identified  one  benchmark,  Unpack,  that  it  called  an 
accepted  industry  standard  for  performance  measurements.  The  paper 


1  Alternative  Control  Parameter  for  High  Performance  Computers. 
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Possible  Ways  of  Safeguarding  U.S.  National 
Security  Interests  Related  to  Computer 
Exports 


noted,  however,  that  changing  the  operating  system  on  a  computer  would 
change  the  measure  and  that  unless  the  exporter  knows  what  operating 
system  the  end  user  has,  the  measure  would  be  impossible  to  accurately 
gauge.  Nonetheless,  the  paper  suggested  continued  study  of  the 
benchmark,  which  might  have  more  relevance  to  higher  performance 
computers  (at  100,000  MTOPS)  in  the  future.  We  observed  that  the 
Commerce  Department's  National  Institute  of  Standards  and  Technology 
does  research  in  the  area  of  computer  systems  performance,  promotes  the 
effective  evaluation  and  efficient  use  of  advanced  computers,  and  designs 
coherent  evaluations  that  economically  and  reliably  characterize  high 
performance  computer  designs. 

Some  computer  industry  representatives  said  that  the  industry  would  never 
achieve  consensus  on  which  benchmarks  to  use.  Furthermore,  they  said 
that  once  any  specific  benchmark  was  established,  computer  designers 
could  circumvent  the  intent  of  the  benchmark. 


Other  Alternatives  to  ln  addition  to  the  three  alternatives  to  replace  MTOPS  considered  by  the 
n  Dn|-srQ  mTHPQ  Dr^xA/n  Commerce-sponsored  advisory  committee,  the  following  two  were 
rvepidLei  I  I  Ur  j  UldWII  identified  from  studies  and  discussions  with  government  and  industry 

From  Studies  and  experts. 

Discussions  With 
Computer  Experts 
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Possible  Ways  of  Safeguarding  U.S.  National 
Security  Interests  Related  to  Computer 
Exports 


I  ndexing  Control  Sandia  National  Laboratory,  computer  industry  representatives,  and  a 

Thresholds  computer  expert  from  academia  have  presented  the  idea  of  indexing  the 

control  thresholds  through  the  use  of  various  criteria,  including  (1)  actual 
sales  or  market  data  or  (2)  a  list  of  the  500  most  powerful  computers, 
which  is  maintained  by  the  University  of  Tennessee  and  the  University  of 
Mannheim.2 

Actual  Sales  According  to  an  analysis  prepared  by  Sandia  National  Laboratory,  one  way 

of  implementing  a  sales-based  index  would  be  for  the  Secretary  of 
Commerce,  in  consultation  with  other  agencies,  to  determine  every 
6  months  or  after  a  petition  by  industry,  which  U.S. -origin  microprocessors 
have  entered  into  mass  production  during  the  preceding  180  days.  The 
Secretary  would  then  have  30  days  to  make  the  determination.  Computers 
incorporating  such  microprocessors  could  then  be  controlled  by  requiring 
export  licenses  based  on  a  multiple  of  the  performance  of  a  single 
mass-produced  microprocessor.  Under  this  indexing  option,  U.S.  computer 
companies  would  have  to  first  establish  a  threshold  level  of  sales  within  the 
United  States  before  they  are  exported,  according  to  Sandia  National 
Laboratory.  The  threshold  might  initially  be  100,  500,  or  more  commercial 
sales  within  the  United  States  and  maybe  tier  1  countries  to  establish  that 
the  product  is  indeed  a  broadly  applicable,  commercial  product.  Once  the 
company  demonstrates  that  it  has  reached  the  threshold  through  actual 
sales,  it  would  no  longer  be  prohibited  from  exporting  such  computers  to 
tier  2  and  tier  3  countries,  although  documentation  of  end  use  and  controls 
could  be  maintained  for  tier  3  countries,  according  to  Sandia  National 
Laboratory. 

Experts  we  consulted  raised  concerns  about  whether  the  sales  or 
market-based  data  would  be  reliable  and  a  U  .S.  government  official  told  us 
that  this  option  was  flawed  because  it  would  not  be  based  on  national 
security  interests.  Finally,  some  computer  industry  representatives 
questioned  whether  this  control  could  be  effectively  implemented  because 


h'o  provide  a  better  basis  for  statistics  on  high  performance  computers,  the  University  of 
Tennessee  and  the  University  of  Mannheim  maintain  a  list  of  sites  that  have  the  500  most 
powerful  computer  systems  installed  in  the  world.  TheTop  500  List  has  been  updated  twice 
a  year  since  J  une  1993  with  the  help  of  high  performance  computer  experts,  computational 
scientists,  manufacturers,  and  the  Internet  community  who  responded  to  a  questionnaire. 
The  producers  of  the  Top  500  List  have  also  used  parts  of  statistical  lists  published  by  others 
for  different  purposes. 
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of  a  perceived  government  inability  to  keep  up  with  market  and/or 
technological  changes. 

Top  500  List 

Another  proposal  suggested  by  some  experts  we  consulted  was  to  apply 
export  controls  to  computers  based  on  the  list  of  the  world's  top  500  most 
powerful  computers  maintained  by  the  U  niversity  of  Tennessee  and  the 
University  of  Mannheim.  For  example,  licenses  could  be  required  for 
computers  whose  capabilities  qualified  them  for  inclusion  on  the  list,  at 
least  at  the  500th  rank.  Alternatively,  the  threshold  could  be  set  for 
computers  surpassing  the  250th,  100th,  or  any  other  ranking  on  the  list. 

Proponents  of  this  idea  point  out  that  the  list  is  based  on  a  transparent 
measure:  a  publicly  available,  testable  benchmark.  Unlike  the  current 
control  system,  this  list  would  not  be  subject  to  sudden  and  significant 
changes  in  license  thresholds  due  to  marketplace  changes  in  just  one  or 
two  chips.  The  controls  would  therefore  be  tied  much  more  closely  to 
technological  realities. 

Other  Ideas  for 
Safeguarding  U.S. 
National  Security 
Interests 

In  addition  to  ideas  to  replace  MTOPS  discussed  above,  thefollowing  seven 
ideas  were  identified  from  studies  and  discussions  with  government  and 
industry  experts  as  other  ways  to  protect  U  .S.  national  security  interests 
relative  to  high  performance  computer  exports. 

Tagging  and  remote  monitoring  is  an  idea  that  has  been  discussed  in  export 
control  literature  for  several  years.  It  is  achieved  by  attaching  a  monitoring 
system  to  the  item  that  is  to  be  exported.  The  active  system  would  both 
monitor  the  object  tagged  and  communicate  that  information  to  the  U  nited 
States.  Elements  of  a  remote  monitoring  system  are  data  acquisition, 
communications,  and  a  command  center. 

U  .S.  government  officials3  and  computer  industry  representatives  pointed 
out  several  limitations  of  tagging  and  remote  monitoring.  Both  procedures 
require  the  cooperation  of  the  end  user  of  the  computer  system.  In 
addition,  the  American  computer  industry  may  resist  having  its  products 


Tagging  and  Remote 
Monitoring 


3These  officials  were  from  the  Lawrence  Livermore,  Los  Alamos  ,  and  Sandia  National 
Laboratories  and  the  Department  of  E  nergy. 
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tagged,  given  that  this  practice  might  discourage  legitimate  foreign 
customers  from  buying  American  computers  because  of  privacy  concerns, 
according  to  the  officials.  They  also  noted  that  a  remote  monitoring  system 
for  exported  computers  might  not  be  feasible  because  of  the  resources  that 
might  be  required  to  monitor  computer  operations  and  send  data  to  the 
United  States.  Also,  even  remote  monitoring  of  computer  operations 
cannot  distinguish  how  the  computer  is  being  used  and  for  what 
applications,  according  to  the  officials.  Some  officials  cautioned  that  any 
type  of  tagging  or  remote  monitoring  to  which  the  end  user  has 
unsupervised  access  could  be  tampered  with  withoutthe  U.S.  government's 
knowledge. 


Assessing  E  nd-U  ser  This  idea  is  simply  a  modification  of  the  current  licensing  process  for 

Attainable  Performance  determining  when  export  control  thresholds  apply.  Under  this  approach, 

the  licensing  threshold  would  be  applied  according  to  the  potential 
capabilities  of  a  computer  system,  rather  than  the  actual  performance  level 
at  the  time  of  sale,  as  under  the  current  system.  One  option  would  be  to 
retain  the  MTOPS  measure  and  adjust  the  control  level  to  account  for  the 
maximum  processing  capability  of  the  system  instead  of  setting  the 
threshold  based  on  the  capabilities  of  the  system  as  configured  at  the  time 
of  shipment.  For  example,  under  the  current  system,  a  computer  that  had 
four  microprocessor  slots  but  was  shipped  with  only  one  microprocessor 
would  likely  be  exported  without  a  license.  However,  the  end  user  could 
buy  three  additional  uncontrolled  microprocessors  and  increase  the  rated 
performance.  Considering  the  end  user's  attainable  performance,  the 
system  would  be  rated  at  its  full  four-processor  potential  and  likely  require 
a  license  to  better  reflect  the  true  capabilities  acquired  in  the  purchase. 
This  standard  could  be  applied  in  any  performance-based  system,  whether 
using  MTOPS  or  another  measure. 

Proponents  of  end-user  attainable  performance  as  a  measure  argue  that 
controls  based  on  maximum  capabilities  prohibit  the  buyerfrom  increasing 
the  capabilities  of  the  purchased  system  without  the  U  .S.  government's 
knowledge.  One  criticism  of  this  approach  is  that  clustering  can  still 
obviate  the  control  because  users  can  link  computers  together  after 
delivery  without  detection  by  the  U.S.  government.  This  control  could  also 
be  rendered  less  effective  or  ineffective  by  advancing  technology  that 
allows  users  to  upgrade  their  computers  with  next  generation  chips, 
making  the  capabilities  of  the  systems  shipped  difficult  or  impossible  to 
track.  In  addition,  some  critics  stated  that  this  control  is  not  replicable  in  a 
consistent  fashion  and  raised  questions  about  how  to  choose  the  control 
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levels  and  use  them.  Finally,  some  doubted  the  acceptability  of  this  option 
within  the  computer  community. 


Raising  Export  Control  Under  this  option,  the  government  could  setthecontrol  levels  to  match  the 

Thresholds  to  the  Level  computational  power  that  can  be  effectively  provided  by  computer  clusters 

0 btained  bv  C I usteri no  composed  of  readily  available  components.  This  approach  to  setting  export 

^  control  levels  is  based  on  the  notion  that  effective  parallel  computers  can 

today  be  built  using  readily  available  commercial  computers  and 
networking  technology  with  performance  between  50,000  to  70,000 
MTOPS.  Periodic  revision  of  the  export  control  levels  could  reflect  the 
combined  effects  of  advances  in  central  processor  unit  technology  and  the 
increasing  capabilities  of  adjunct  technologies  in  communications  and 
networking  to  achieve  very  high  levels  of  computing  power  by  linking 
together  larger  numbers  of  computers. 

Using  this  approach,  today's  large  inconsistency  between  the  levels  at 
which  high  performance  computers  are  controlled  and  the  performance 
levels  achievable  by  commercially  available  parallel  clusters  would  be 
eliminated,  according  to  the  Lawrence  Livermore  National  Laboratory 
analysis.4  The  analysis  stated  that  the  true  state  of  U  .S.  exposure  to 
changing  national  security  threats  would  be  acknowledged  and  addressed. 
Appropriately  responding  to  the  implications  of  the  actual  availability  of 
high  performance  computing  would  require  extensive  review  and 
discussion  in  the  national  security  community.  Some  critics  of  this 
approach  stated  that  this  control  is  not  replicable  in  a  consistent  fashion 
and  raised  questions  about  how  to  choose  the  control  levels  and  use  them. 


4The  National  Security  Implications  of  Decontrolling  Export  to  Tier  III  Countries  of  High 
Performance  Computers  between  2,000  and  40,000  MTOPS  (Feb.  2000). 
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Controlling  Software  This  option  involves  denying  countries  of  concern  key  software 

Applications  applications  used  in  high  performance  computing  through  the  use  of 

r  security  classifications  and/or  export  controls.  According  to  the  Director  of 

DOD's  High  Performancecomputing  Modernization  Program,5  one  concept 
under  discussion  is  the  development  of  approaches  to  ensure  that  software 
is  used  as  intended  and  authorized.  He  said  that  research  would  be  needed 
to  develop  encryption  technology  to  restrict  access  for  source  and 
executable  applications.  The  intent  of  this  concept  is  to  develop  means  that 
will  prevent  militarily  relevant  applications  from  operating  on  unauthorized 
machines.  The  official  stated  that  technology  advances  can  provide 
improved  protections  for  militarily  relevant  applications.  According  to  the 
DOD  Deputy  U  nder  Secretary  (Science  &  Technology),  militarily  significant 
software  might  be  made  more  tamper-proof,  that  is,  able  to  operate  on  only 
one  type  of  machine  or  to  self-destruct  if  tampered  with.  The  official  said 
that  DOD  is  just  beginning  to  examine  this  option. 

The  prevalent  view  among  government  and  computer  industry  officials 
with  whom  we  spoke  is  that  the  applications  software  that  can  be 
controlled  is  already  protected  by  security  classification,  export  controls, 
and  proprietary  rights  and  that  commercial  dual-use  software  cannot  be 
controlled.  A  Lawrence  Livermore  National  Laboratory  analysis  pointed 
out  that  some  software  used  in  clusters  is  freely  available  on  the  Internet. 
Operating  systems  such  as  Linux  can  also  be  downloaded  free  from  the 
Web.  Even  operating  systems  like  Windows  NT  or  Windows  2000  are  sold  in 
such  quantities  that  controls  are  viewed  as  impractical  and  unenforceable. 
Research  and  academic  institutions  whose  culture  is  characterized  by  the 
free  exchange  of  information  and  software  are  notable  users  and 
developers  of  high  performance  computing  technology.  Thus,  it  is  difficult 
to  find  any  part  of  the  software  application  market  that  lends  itself  to 
controls  because  distribution  is  easy  and  applications  of  national  interest 
are  already  tightly  held  by  their  owners,  either  through  security 
classification  or  efforts  to  protect  a  commercial  interest.  In  contrast,  the 
Director  of  DOD's  High  Performance  Computing  Modernization  Program 
stated  that  dual-use  applications  are  being  controlled  to  a  limited  extent. 


5The  DOD  High  Performance  Computing  Modernization  Program  provides  advanced 
hardware,  computing  tools,  and  training  to  DOD  researchers  utilizing  the  latest  technology 
to  aid  their  mission  in  support  of  the  warfighter.  The  program  seeks  to  modernize  the  total 
high  performance  computational  capability  of  DOD  Science  and  Technology,  Development 
Test  and  Evaluation,  and  Ballistic  Missile  Defense  Organization  to  a  level  comparable  to  that 
available  in  the  foremost  civilian  and  other  government  agency  research  and  development 
environments. 
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He  noted  that  several  commercially  available  applications  cannot  be  run 
without  receiving  an  annually  provided  key  from  the  vendor. 


Controlling  Technology  The  idea  of  controlling  technology  used  to  link  computers  together  is  based 

Used  for  Interconnection  on  the  control  of  the  number  and  capability  of  commercial  readily  available 

parallel  clusters  that  can  be  created  by  controlling  networking  hardware 
such  as  switches,  interface  cards,  and  related  equipment. 

However,  the  hardware  used  in  high-speed,  commercial,  ready-made 
networking  is  already  widely  available  and  relatively  inexpensive 
throughout  the  world,  according  to  U.S.  government  officials6  and 
computer  industry  representatives.  They  believe  that  establishing  controls 
at  this  point  would  be  extremely  difficult,  if  not  impossible.  To  be  effective, 
this  approach  would  depend  on  networking  technology  originating  solely  in 
the  United  States  or  countries  with  binding  agreements  to  enforce  U.S. 
export  control  objectives,  according  to  U  ,S.  government  officials  and 
computer  industry  representatives.  However,  manufacturing  capability  for 
interconnecting  technology  is  available  worldwide,  and  with  the  growth  in 
the  cluster  market,  it  will  be  considerably  more  difficult  to  restrict  supply. 
Industry  representatives  consider  only  some  of  the  equipment  related  to 
this  technology  controllable  at  the  higher  end  of  the  market  because  of  its 
proprietary  nature  and  the  continued  importance  of  vendor  support. 


Controlling  Computer 
Systems  Based  on 
Bandwidth 


According  to  the  Director  of  DOD's  High  Performance  Computing 
Modernization  Program,  a  concept  under  discussion  would  develop  a 
methodology  to  control  computer  systems  by  some  measure  of  processor- 
to-main-memory  bandwidth7  and  potentially  the  number  of  processors  in 
each  system.  The  methodology  would  need  to  distinguish  between 
commodity  systems  and  the  traditional  class  of  supercomputers 
characterized  by  specialized  processors.  Such  computer  systems  typically 
have  a  single  processor  and  require  a  high  level  of  bandwidth,  such  as  those 
manufactured  by  Cray,  NEC,  and  Fujitsu.  Another  official  from  DOD's  High 
Performance  computing  Modernization  Program  said  that  these  computers 
still  possess  capabilities,  capacities,  and  characteristics  that  should  be 


6  Officials  were  from  the  Lawrence  Livermore,  Los  Alamos,  and  Sandia  National 
Laboratories  and  the  Departments  of  Commerce,  Defense,  and  E  nergy. 

7Bandwidth  is  the  amount  of  data  that  can  be  transmitted  in  a  fixed  amount  of  time. 
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denied  to  countries  of  concern.  The  DOD  official  identified  as  a  concern 
those  computers  that  can  solve  militarily  significant  problems  that  cannot 
be  broken  down  and  operated  on  clustered  systems.  Computer  industry 
experts  placed  the  total  market  for  these  computers  in  the  range  of  300  per 
year.  The  Department  did  not  provide  any  additional  information  about  this 
option  because  it  was  still  under  review. 


Implementing 
Countermeasures  to 
Military  Advantages  Gained 
by  Countries  of  Concern 
From  More  Advanced 
Computer  Exports 


Using  this  option,  DOD  would  design  countermeasures  to  deal  with  the 
implications  of  wider  computer  availability.  One  purpose  of  export  controls 
is  to  maintain  the  U.S.  technological  lead  in  military  capabilities  by  denying 
transfers  of  advanced  technology  to  countries  of  concern.  As  technological 
advances  in  high  performance  computing  make  this  purpose  more  difficult 
for  export  controls  to  achieve,  it  may  become  necessary  to  explore  other 
options  to  maintain  the  U.S.  technology  lead.  As  a  step  in  this  direction, 
section  1402  of  the  National  Defense  Authorization  Act  for  Fiscal  Year  2000 
requires  an  annual  report  on  transfers  of  militarily  sensitive  technology  to 
countries  and  entities  of  concern.  It  is  to  include  an  assessment  by  the 
Secretary  of  Defense,  in  consultation  with  the  J  oint  Chiefs  of  Staff  and  the 
Director  of  Central  Intelligence,  on  the  cumulative  impact  of  U.S. -granted 
licenses  for  exports  of  technologies  and  technical  information.  This  report 
is  required  to  include  information  on  countermeasures  that  may  be 
necessary  to  overcome  the  use  of  such  technologies  and  technical 
information.  It  is  not  required  to  include  an  assessment  of  the  cumulative 
impact  of  nonlicensed  computers,  such  as  those  that  could  be  clustered. 
The  report  is  scheduled  for  completion  in  late  2000. 
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In  the  past  few  years,  broadly  recognized  technological  advances  in  the 
ability  to  cluster  together  low  performance  computers  have  resulted  in  an 
increasing  capability  worldwide  to  achieve  high  performance  computer 
levels.  As  a  result,  the  known  performance  levels  of  clustered  systems 
exceed  the  export  control  thresholds  set  by  the  U  .S.  government.  I  n 
addition,  U.S.  government  officials1  and  computer  industry  representatives 
have  discussed  the  shortcomings  of  the  MTOPS  measure  for  export  control 
purposes  and  now  agree  that  MTOPS  is  no  longer  a  valid  measure  of 
computer  performance  and  needs  to  be  replaced. 


Computer  Advances 
Clustered  Systems 


j  p  Technological  advances  in  computing  that  would  threaten  the  effectiveness 
of  export  controls  have  been  expected  for  several  years.  For  example,  a 
1994  U.S.  government  study2  warned  that  technological  computing 
advances  would  seriously  impair  the  ability  of  U.S.  export  controls  to 
protect  computer  technology  by  the  turn  of  the  century.  A  Defense-  and 
Commerce-commissioned  team  in  19953  reviewed  the  computer  industry's 
technological  advances  in  parallel  processing  and  concluded  that  advances 
in  clustering  contribute  to  the  uncontrollability  of  high  performance 
computing  worldwide  and  inevitably  reduce  the  effectiveness  of  U.S. 
export  controls.  One  of  the  technology  trends  of  concern  at  that  time 
included  other  countries'  ability  to  link  individual  computers  to  achieve 
higher  performance  levels.  We  reported  2  years  ago  that  trends  in  high 
performance  computing  technology  development  might  pose  security  and 
export  control  challenges  and  recommended  further  study  to  determine 
their  implications  for  national  security  and  export  controls.4 


According  to  a  Lawrence  Livermore  National  Laboratory  analysis  dated 
February  2000,  in  determining  the  usefulness  of  clusters  made  with  readily 
available  commercial  components,  the  following  four  aspects  of  computing 
must  be  considered: 


1  U.S.  government  officials  include  the  Deputy  Under  Secretary  of  Defense  (Science  and 
Technology)  and  officials  of  the  Commerce  and  E  nergy  Departments. 

2  The  unclassified  title  of  this  classified  report  is  High  Performance  Computing  Technology: 
Implications  for  Development  of  Weapons  of  Mass  Destruction. 

3  High-Performance  Computing  Export  Control  Policy  in  the  1990s ( Palo  Alto,  CA:  Stanford 
University,  Center  for  International  Security  and  Arms  Control,  1995). 

4  Export  Controls  (GAO/NSIAD-98-196,  Sept.  16, 1998). 
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•  Performance,  flexibility,  low  cost,  and  the  ease  of  integration,  taken 
together,  make  the  development  of  effective  parallel  computers  using 
commercial  readily  available  networking  hardware  dramatically  easier 
than  it  was  just  3  years  ago,  according  to  Lawrence  Livermore  National 
Laboratory  officials.  Commercial  clusters  of  readily  available 
components  perform  up  to  about  a  third  as  well  as  today's  best 
distributed  memory  parallel  computers  (stand-alone  high  performance 
computers),  while  the  cost  of  clustering  adds  only  10  to  30  percent  to 
the  cost  of  the  computers  that  are  clustered. 

•  The  system  software  necessary  to  integrate  a  cluster  of  computers  into 
an  effective  computing  system  with  a  correspondingly  higher  capability 
than  its  individual  constituent  processors  is  now  widely  available. 

•  Programming  a  commercial  cluster  of  readily  available  components  is 
no  different  from  programming  most  other  commercial  parallel 
processing  computers  because  the  system  software  is  the  same  on  both. 
Computer  science  departments  in  universities  worldwide  teach 
concepts  to  support  the  development  of  parallel  applications  for 
commercial,  readily  available  clusters.  Other  U  .S.  government  and 
computer  industry  specialists,  however,  stated  that  there  might  be  some 
number  of  computer  applications  that  cannot  be  adapted  for  clustered 
parallel  processing.  To  the  extent  that  such  applications  are  militarily 
significant  and  must  be  operated  on  individual  computers,  maintaining 
threshold  controls  on  these  machines  could  still  be  effective. 

•  Support  for  computer  functions  means  having  an  efficient  schedule  for 
running  hundreds  of  separate  problems  and  the  input  and  output  of  data 
among  a  computer,  its  local  disks,  networks,  and  archival  storage. 
Without  vendor-supplied  software  to  automate  these  functions, 
everything  must  be  done  manually,  making  production  computing  a 
labor-intensive  operation.  While  U.S.  practice  is  to  pay  computer 
vendors  to  supply  this  support,  foreign  countries  such  as  India  and 
China,  where  skilled  labor  is  plentiful  and  low  cost,  may  find  providing 
this  support  much  less  of  a  problem. 

Other  U.S.  government  and  computer  industry  specialists  also  stated  that 
there  might  be  some  computer  applications  that  cannot  be  adapted  for 
clustered  parallel  processing.  The  President's  August  30,  2000,  report 
indicated  that  some  small  number  of  computer  applications  still  requires 
traditional  computer  systems  and  that  the  lack  of  hardware  may  be  a 
barrier  to  solving  certain  kinds  of  problems.  The  President's  report  did  not 
identify  these  applications.  However,  computer  applications  that  have 
problems  which  cannot  be  broken  up  to  be  solved  in  parallel  and 
applications  associated  with  battlefield  operations  that  rely  on  rapid 
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solutions  might  not  practically  be  run  on  clustered  computer  systems, 
according  to  ISTAC  members  and  a  DOD  official. 


Shortcomings  of 
MTOPS  as  a  Measure 
of  Performance 


U.S.  government  officials5  and  computer  industry  representatives  agreed 
that  MTOPS  is  no  longer  a  valid  measure  of  computer  performance  and 
needs  to  be  replaced.  In  1991,  the  Commerce  Department  began  using 
MTOPS  as  a  measure  to  decide  when  an  export  license  would  be  required 
for  high  performance  computers.  But  because  of  the  diverse  computer 
designs  that  have  evolved  since  then,  the  increasing  performance  level  of 
commercial  microprocessors,  and  the  capability  to  connect  local  and  wide 
area  networks,  it  became  prudent  to  reexamine  the  suitability  of  MTOPS  as 
a  measure,  according  to  the  I  nstitute  for  Defense  Analyses.  As  a  result, 
under  DOD  auspices,  U.S.  government  officials,  computer  industry 
representatives,  and  academics  met  in  December  1997  to  determine 
whether  the  use  of  MTOPS  is  sufficient  to  rate  the  relative  performance  of 
current  and  future  computer  systems.  They  determined  the  following: 


•  The  use  of  MTOPS  was  still  an  effective  means  at  that  time  for 
determining  export  controls  for  a  single  computing  element.  The 
attendees  stated  that  modest  refinements  could  be  made  to  the  MTOPS 
measure  for  systems  comprising  aggregate  computing  elements. 

•  There  could  be  variances  up  to  a  factor  of  2  in  the  actual  performance  of 
delivered  systems  relative  to  the  MTOPS-calculated  performance 
because  of  the  wide  range  of  designs  in  use.  Continuing  rapid  changes  in 
technology  might  result  in  yet  larger  variances  in  the  near  future. 

•  Any  export  control  measure  should  be  reevaluated  every  2  years 
because  of  the  rapid  changes  in  computer  designs. 

By  2000,  however,  it  had  become  clear  to  both  government  and  industry 
computer  experts  that  MTOPS  does  not  account  for  new  processor 
designs,  particularly  those  dependent  on  the  capability  to  exchange 
information  internally  at  high  speeds.  Representatives  of  three  computer 
companies  said  that  MTOPS  was  an  archaic  measure  not  designed  with 
new  chip  designs  in  mind  and  should  be  replaced.  The  MTOPS  measure 
today  is  unreliable  because  it  does  not  account  for  certain  factors,  such  as 
memory  access  times,  interconnection  methods,  and  internal  bus  speeds. 
As  a  result,  two  microprocessors  with  the  same  megahertz  could  have 


5  U.S.  government  officials  include  the  Deputy  Under  Secretary  of  Defense  (Science  and 
Technology)  and  officials  of  the  Commerce  and  E  nergy  Departments. 
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different  MTOPS  ratings  if  the  additional  factors  were  considered.  The 
MTOPS  indicator  also  would  be  inaccurate  if  it  were  applied  incorrectly  for 
computing  elements  organized  in  certain  ways,  such  as  in  clusters  of 
individual  computers.  It  is  not  accurate  to  take  the  MTOPS  measure  of  one 
clustered  machine  and  multiply  it  by  the  total  number  of  computing 
processors  of  the  cluster  because  key  characteristics,  such  as  the  time 
taken  to  distribute  tasks  and  exchange  intermediate  results,  have  not  been 
taken  into  account. 
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This  appendix  lists  the  applications  of  national  security  importance 
identified  for  the  1998  Defense-  and  Commerce-sponsored  study,  which 
was  the  basis  for  information  on  computer  applications  in  the  President's 
reports  in  February  and  August  2000.  The  list  is  reprinted  in  its  entirety  as  it 
appeared  in  the  1998  study. 
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Appendix  A:  Applications  of  National  Security  Importance 

The  following  table  provides  a  summary  of  national  security  applications  reviewed  in  this 
study.  It  indicates  the  kind  of  problem  solved,  the  HPC  configuration  on  which  it  was  solved, 
and  the  time  required  for  the  solution. 

This  selection,  compiled  through  a  combination  of  direct  communications  with 
practitioners  and  a  review  of  published  literature,  is  not  an  exhaustive  listing.  However,  it  does 
include  many  of  the  more  important  national  security  applications,  and  gives  policy  makers  a 
rough  idea  of  the  kinds  of  applications  being  solved  at  various  performance  levels. 

Two  points  in  particular  should  be  kept  in  mind  when  reading  the  table.  First,  the 
applications  shown  here  constitute  data  points  that  often,  in  practice,  lie  along  a  continuum. 
The  specific  size  of  the  application  is  often  a  function  of  the  computational  resources  available 
in  a  given  configuration  and  the  “threshold  of  patience”  of  the  practitioner.  If  the  configuration 
available  were  slightly  larger,  or  smaller,  the  practitioners  in  most  cases  would  solve  the  same 
kinds  of  problem,  but  perhaps  with  a  different  grid  size,  or  time-step,  etc.  In  short,  the 
absence  of  a  particular  type  of  application  at  some  performance  level  should  not  be 
interpreted  as  a  statement  that  no  version  of  that  application  can  be  solved  at  that  performance 
level. 

Second,  the  CTP  value  shown  is  the  composite  theoretical  performance  of  the 
configuration  used  to  solve  the  problem.  It  is  well  known  that  any  metric,  including  the  CTP, 
does  not  perfectly  predict  the  performance  of  all  systems  on  all  applications.  Consequently, 
the  CTP  measures  given  here  should  be  used  only  as  rough  indicators  of  the  performance  level 
required  for  a  particular  kind  of  application.  The  fact  that  a  given  problem  was  run  on  a 
machine  with  a  CTP  of  n  Mtops  does  not  mean  that  all  systems  with  CTP  >  n  Mtops  can  solve 
the  problem,  or  that  all  systems  with  CTP  <  n  Mtops  can  not.  The  CTP  simply  does  not  have 
this  kind  of  precision. 

The  following  acronyms  are  used  for  applications  categories: 


CCM 

Computational  Chemistry  and 
Materials  Science 

CWO 

Climate/Weather/Ocean 

Modeling  and  Simulation 

CEA 

Computational 

Electromagnetics  and 

Acoustics 

FMS 

Forces  Modeling  and 
Simulation/C4I 

CFD 

Computational  Fluid 

Dynamics 

Nuclear 

Nuclear  weapons  development 
and  stockpile  maintenance 

CSM 

Computational  Structural 
Mechanics 

SIP 

Signal/Image  Processing 
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Machine 

Year 

CTP 

Category 

Time 

Problem 

Problem  size 

VAX  6210 

1 

SIP 

35  min 

Focus  an  image  [1] 

5040  x  1260  samples 
(18km  x  8km) 

Cray-1 

1984 

195 

CFD 

1.4  CPU 
hours 

SCRAMJET  wing-fuselage  aerodynamic 
interaction  simulation  [2] 

56,730  grid  points, 

Mach  6 

Cray- IS 

1984 

195 

CFD 

20  CPU  hr 

Simulation  of  after-body  drag  for  a 
fuselage  with  propulsive  jet.  Reynolds 
averaged  Navier- Stokes  [3] 

Cray- IS 

early 

1980s 

195 

nuclear 

1127.5 

CPU  sec 

LANL  Hydrodynamics  code  3  [4] 

Cray- IS 

early 

1980s 

195 

nuclear 

117.2  CPU 

sec 

LANL  Hydrodynamics  code  1  [4] 

Cray- IS 

early 

1980s 

195 

nuclear 

4547.1 

CPU  sec 

LANL  Hydrodynamics  code  2  [4] 

Cray-1 

195 

{ 

24  hours 

Crash  simulation  [5] 

5,500  elements 

Cray- IS 

195 

CFD 

Nonlinear  Inviscid  (STAGE  II):  Above, 
plus  transonic  pressure  loads;  wave  drag 
13,6} 

100,000  grid  points 

Cosmic  Cube  (6) 

1991 

293 

SIP 

1.81 

millisec 

Discrete  Fourier  transform  algorithm  [7] 

5040  complex  data 
sample 

Cray  X-MP/1 

mid 

1980s 

316 

nuclear 

Two-dimensional,  reduced  physics 
simulation 

Cray  XMP/48 

(Iproc) 

1988 

353 

CFD 

20-50  hr 

Flow  simulation  around  complete  F-16A 
aircraft  (wings,  fuselage,  inlet,  vertical 
and  horizontal  tails,  nozzle)  at  6  deg  angle 
of  attack,  Mach  0.9.  Reynolds  Averaged 
Navier- Stokes.  Reynolds  number  =  4.5 
million.  [8] 

1  million  grid  points,  8 
Mwords  (one  2  Mword 
zone  in  memory  at  a 
time),  2000-5000 
iterations 

Cray  XMP/1 

1990 

353 

CCM 

1000  CPU 
hours 

Molecular  dynamics  of  bead-spring  model 
of  a  polymer  chain  (9] 

Chain  length=400 

Cray  J916/1 

1996 

450 

CFD 

300  CPU 
hr 

Modeling  of  transonic  flow  around 

AS28G  wing/body/pylon/nacelle 
configuration.  3-D  Reynolds  averaged  full 
Navier- Stokes  solution.  [10] 

3.5  million  nodes,  195 
Mwords  memory 

0rigin2000/l 

1997 

459 

SIP 

5.650  s 

RT.STAP  benchmark  (harJuTi] 

2.7  million  samples  per 
.161  sec 

Cray  Y-MP/1 

1987 

50C 

CSM 

200  hours 

3-D  shock  physics  simulation  [12] 

200,000  cells 

Cray  YMP/1 

1990 

500 

CSM 

39  CPU 

sec 

Static  analysis  of  aerodynamic  loading  on 
solid  rocket  booster  [13] 

10,453  elements, 9206 
nodes, 54,870  DOF  256 
Mword  memory 

Cray  YMP 

1991 

500 

CCM 

1000  CPU 
hours 

Molecular  dynamics  modeling  of 
hydrodynamic  interactions  in  “semi¬ 
dilute”  and  concentrated  polymer 
solutions  [9] 

single  chain,  60 
monomers 

Cray  YMP 

1991 

500 

CCM 

1000  CPU 
hours 

Modeling  of  thermodynamics  of  polymer 
mixtures  [9] 

lattice  size  - 
112EXP(3);  chain  size 
=  256 

Cray  YMP/1 

1991 

500 

CFD 

40  CPU  h 

Simulation  of  viscous  flow  about  the 
Harrier  Jet  (operating  in-ground  effect 
modeled)  [14] 

2.8  million  points,  20 
Mwords  memory. 

Cray  YMP/1 

1993 

500 

CCM 

1.47  CPUs/ 
timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid-state  point  [15] 

100,000  atoms 

Cray  YMP/1 

1996 

500 

CFD 

170  CPU 
hr  (est) 

Modeling  of  transonic  flow  around 

AS28G  wing/body/pylon/nacelle 
configuration.  3D  Reynolds  Averaged  full 
Navier-Stokes  solution  [10] 

3.5  million  nodes,  195 
Mwords  memory 

Cray  YMP/1 

1996 

500 

CFD 

6  CPU  hr 

Modeling  of  transonic  flow  around  F5 
wing  (Aerospatiale).  3D  Reynolds 

Averaged  full  Navier-Stokes  solution  [10] 

442368  cells  — 

(192x48x48). 

Cray  YMP 

1996 

500 

CFD 

8  CPU  hr, 

3000 

timesteps 

Large-Eddy  simulation  at  high  Reynolds 
number  [16] 

2.1  million  grid  points, 
44  Mwords 
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Machine 

Year 

|  CTP 

Category 

Time 

Problem 

Problem  size 

workstation 

1997 

500 

CSM 

2-D  modeling  of  simple  projectile  striking 
simple  target.  [17] 

10,000s  of  grid  points 

Cray  Y-MP/1 

late 

1980s 

500 

nuclear 

1000s 

hours 

two-dimensional,  almost  full  physics  (e.g. 
Monte  Carlo  neutron  transport) 

Cray  Y-MP/1 

late 

1980s 

500 

nuclear 

seconds 

one-dimensional,  full  physics  simulation 

Cray  YMP/1 

500 

CEA 

5  CPU  hr 
per 

timestep 

Signature  of  modern  fighter  at  fixed 
incident  angle  at  1  GHz  [18] 

50  million  grid  points  @ 
18  words/grid  point 

Cray  Y-MP/1 

500 

nuclear 

two-dimensional,  reduced  physics 
simulations 

100-500  Mbytes 

Mercury  Race  (5 
x  4  i860 
processors, 
Ruggedized) 

1997 

866 

SIP 

SAR  system  aboard  P3-C  Orion  maritime 
patrol  aircraft  [19] 

Cray  YMP/2  256 
MW 

1990 

958 

CSM 

19.79  CPU 

s 

Static  analysis  of  aerodynamic  loading  on 
solid  rocket  booster  [13] 

10,453  elements,  9206 
nodes,  54,870  DOF 

Cray  Y-MP/2 

late 

1980s 

958 

CFD 

Design  of  F-22  fighter  [20] 

CM-5/32 

1993 

970 

CCM 

449  CPU 

sec 

Determination  of  structure  of  Eglin-C 
molecular  system  [21] 

530  atoms,  with  1689 
distance  and  87 
dihedral  constraints 

Cray-2/1 

1987 

1,098 

CSM 

400  hours 

3D  modeling  of  projectile  striking  target. 
Hundreds  of  microsecond  time  scales. 

[17] 

.5-1.5  million  grid 
points256  Mword 
memory 

Cray-2 

1992 

1,098 

CSM 

5  CPU 
hours 

Modeling  aeroelastic  response  of  a 
detailed  wing-body  configuration  using  a 
potential  flow  theory  [13] 

Cray-2 

1992 

1,098 

CSM 

6  CPU 
days 

Establish  transonic  flutter  boundary  for  a 
given  set  of  aeroelastic  parameters  [13] 

Cray-2 

1992 

1,098 

CSM 

600  CPU 
days 

Full  Navier-Stokes  equations  [13] 

Cray-2 

1,098 

CSM 

2  hours 

3-D  modeling  of  symmetric,  transonic, 
low  angle  of  attack  impact  of  warhead 
and  defensive  structure  [20] 

Cray- 2 

1,098 

CSM 

200  hours 

Penetration  model  against  advanced 
armor  [20] 

Cray-2 

1,098 

CSM 

2000 

hours 

Modeling  full  kinetic  kill  effects  against 
hybrid  armors  [20] 

Cray- 2 

1,098 

CSM 

40  hours 

3-D  modeling  of  asymmetric,  transonic, 
low  angle  of  attack  impact  of  warhead 
and  defensive  structure  [20] 

Cray-2 

1984 

1,098 

CFD 

15  CPU  m 

Simulation  of  2-D  viscous  flow  field 
about  an  airfoil  [3] 

Cray-2 

1988 

1,098 

CFD 

20  hr 

Simulation  of  flow  about  the  space  shuttle 
(Orbiter,  External  Tank,  Solid  Rocket 
Boosters),  Mach  1.05,  Reynolds  Averaged 
Navier-Stokes,  Reynolds  number  =  4 
million  (3%  model)  [8] 

750,000  grid  points,  6 
Mwords. 

Cray-2 

1980s 

1,098 

CFD 

100  CPU  h 

Simulation  of  external  flow  about  an 
aircraft  at  cruise.  Steady  flow.  Steady 
Navier-Stokes  simulation.  [14] 

1.0  million  grid  points. 

1995 

1,400 

nuclear 

Credible  one-  and  two-dimensional 
simulations  [22] 

Cray  C90/1 

1993 

1,437 

CCM 

.592  sec/ 
timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

100,000  atoms 

Cray  C90/1 

1994 

1,437 

C^A 

161  CPU 
hour 

Compute  magnitude  of  scattered  wave- 
pattern  on  X24C  re-entry  aerospace 
vehicle  [23] 

181x59x162  grid  (1.7 
million) 

Cray  C90/1 

1994 

1,437 

CFD 

overnight 

Modeling  of  flow  over  a  submarine  hull 
with  no  propulsion  unit  included  [24] 

1-2  million  grid  points. 
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Year 

CTP 

Category 

Time 

Problem 

Problem  size 

Cray  C916 

1994 

1,437 

CEA 

Radar  cross  section  on  perfectly 
conducting  sphere  [25] 

48x48x9 6  ((221 
thousand) 

Cray  C90/1 

1995 

1,437 

CEA 

1  hour 

Submarine  acoustic  signature  for  single 
frequency 

Cray  C90/1 

1995 

1,437 

CEA 

1  hour 

Submarine  acoustic  signature  for  single 
frequency  [26] 

Cray  C90/1 

1995 

1,437 

CEA 

12,475  sec 

Radar  cross  section  of  perfectly 
conducting  sphere,  wave  number  20  [27] 

97x96x192  (1.7  million 
grid  points),  16.1  points 
per  wavelength 

Cray  C90/1 

1995 

1,437 

CEA 

200  hour 

Submarine  acoustic  signature  for  full 
spectrum  of  frequencies  [26] 

Cray  C90/1 

1995 

1,437 

CWO 

CCM2,  Community  Climate  Model,  T42 
[28] 

128  x  64  transform 
grid,  4.2  Gflops 

Cray  C90/1 

1996 

1,437 

CpD 

19  CPU  hr 

Simulation  of  turbulent  flow  around  the 
F/A-18  aircraft  at  60  degree  angle  of 
attack.  Mach  0.3.  Reynolds  number  5= 

8.88  million  [29] 

1.25  million  grid 
points,  100  Mwords  of 
memory. 

Cray  C90/1 

1996 

1,437 

CFD 

200  CPU 
hr 

Simulation  of  unsteady  flow  about  an  F-18 
High  Alpha  Research  Vehicle  at  30,  45, 

60  deg  angle  of  attack  [30] 

2.5  million  grid  points 
for  half-body  modeling. 

40  Mwords  memory 

Cray  C90/1 

1996 

1,437 

CFD 

3  CPU  hr 

Modeling  of  flow  over  a  blended 
wing/body  aircraft  at  cruise.  [31] 

45  Mwords  of  memory 

Origin2000/4 

1997 

1,517 

SIP 

1.475  s 

RT_STAP  benchmark  (hard)  [11] 

2.7  million  samples  per 
.161  sec 

SGI 

PowerChallenge4 

nodes 

1997 

1,686 

CCM 

overnight 

Explosion  simulation  [32] 

30  thousand  diatomic 
molecules 

SGI  Onyx 

1990 

1,700 

SIP 

Attack  and  Launch  Early  Reporting  to 
Theater  (ALERT)  [20]. 

Mercury  Race 

(52  processor, 
i860) 

1996 

1,773 

SIP 

Sonar  system  for  Los  Angeles  Class 
submarines  [33] 

Cray  YMP/4  256 
MW 

1990 

1,875 

CSM 

10  CPUs 

Static  analysis  of  aerodynamic  loading  on 
solid  rocket  booster  [13] 

10,453  elements,  9206 
nodes,  54,870  DOF 

Intel  iPSC/860/64 

1993 

2,097 

CCM 

.418 

CPUs/time 

step 

MD  simulation  using  short-range  forces 
model  applied  to  3D  configuration  of 
liquid  near  solid  state  point  [15] 

100,000  atoms 

Intel  iPSC/8  60/64 

1993 

2,097 

CCM 

3.68  CPUs/ 
timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3D  configuration  of 
liquid  near  solid  state  point  [15] 

1  million  atoms 

Cray  2,  4  proc 

1990 

2,100 

CSM 

400  hours 

armor/anti-armor,  3-D 

Cray  T3D/16 

1996 

2,142 

CFD 

20,000 

CPU  sec. 

50  CPU 
sec  x  400 
steps 

Aerodynamics  of  missile  at  Mach  3.5. 
Reynolds  number  =  6.4  million  [34] 

500x150  (75,000) 
elements  in  mesh. 
381,600  equations 
solved  every  timestep. 

CM-2 

2,471 

SIP 

10  minutes 

Creation  of  synthetic  aperture  radar 
image  [20] 

Cray  C90/2 

1993 

2,750 

CCM 

117  CPU 
sec 

Determination  of  structure  of  Eglin-C 
molecular  system  [21] 

530  atoms,  with  1689 
distance  and  87 
dihedral  constraints 

Cray  C90/2 

1993 

2,750 

CCM 

12269  sec 

Determination  of  structure  of  E.  coli  trp 
repressor  molecular  system  [21] 

1504  atoms6014 
constraints 

iPSC860/128 

1997 

3,485 

CFD 

120  hr 

small  unmanned  vehicle,  fully  turbulent 

iPSC/860  128 

nodes 

1997 

3,485 

CFD 

5  days 

Full  ship  flow  model  [35] 

iPSC860/128 

1997 

3,485 

CFD 

5  days 

Small  unmanned  undersea  vehicle.  Fully 
turbulent  model  with  Reynolds  numbers. 
[35] 

2.5  million  grid  points 

Cray  YMP/8 

1989 

3,708 

CFD 

Model  of  flow  around  a  fully  appended 
submarine.  Steady  state,  non -viscous  flow 
model.  [35] 

250,000  grid  points 

Cray  Y-MP/8 

early 

1990s 

3,708 

CSM 

200  hours 

3D  shock  physics  simulation  [12] 

6  million  cells 
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Problem  size 

Cray  Y-MP/8 

mid 

1990s 

3,708 

CSM 

10-40  hrs 

3-D  shock  physics  simulation  [12] 

100K-1  million  cells 

IBM  SP1/64 

1993 

4,074 

CCM 

1.11  sec 
timestep 

Molecular  dynamics  of  Si02  system  [36] 

0.53  million  atoms 

Intel  Paragon 

4,600 

CEA 

Non-acoustic  anti-submarine  warfare 
sensor  development  [20] 

IBM  SP-2/32 

1996 

4,745 

CFD 

80  minutes 

3-D  unsteady  incompressible  time- 
averaged  Navier-Stokes.  Multiblock 
transformed  coordinates  [37] 

3.3  million  points 

IBM  SP-2/32 

1997 

4,745 

CFD 

Helicopter  rotor  motion  coupled  with 
rotor  CFD,  predict  3-D  tip-relief  flow 
effect,  parallel  approximate  factorization 
method  [38] 

0rigin2000/12 

1997 

4,835 

CFD 

4329  sec 

CFL3D  applied  to  a  wing-body 
configuration:  time-dependent  thin-layer 
Navier-Stokes  equation  in  3-D,  finite- 
volume,  3  multigrid  levels  [39] 

3.5  million  points 

Intel  Paragon/150 

early 

1990s 

4,864 

CFD 

JAST  aircraft  design  [20] 

Cray  C90/4 

1993 

5,375 

CFD 

1  week 

Modeling  of  flow  around  a  smooth 
ellipsoid  submarine.  Turbulent  flow,  fixed 
angle  of  attack,  [35] 

2.5  million  grid  points 

CM-5/128 

1993 

5,657 

CCM 

436  CPU 

sec 

Determination  of  structure  of  Eglin-C 
molecular  system  [21] 

530  atoms,  with  1689 
distance  and  87 
dihedral  constraints 

CM-5/128 

1993 

5,657 

CCM 

6799  CPU 

sec 

Determination  of  structure  of  E.  coli  trp 
repressor  molecular  system  [21] 

1504  atoms,  with  6014 
constraints 

0rigin2000/16 

1997 

5,908 

SIP 

.39  s 

RT_STAP  benchmark  (hard)  [11] 

2.7  million  samples  per 
.161  sec 

IBM  SP-2/45 

1997 

6,300 

CFD 

2  to  4 
hours 

Helicopter  blade  structural  optimization 
code,  gradient- based  optimization 
technique  to  measure  performance 
changes  as  each  design  variable  is  varied 
[38] 

up  to  90  design 
variables,  one  run  per 
variable 

Cray  T3D/64 

1995 

6,332 

CWO 

CCM2,  Community  Climate  Model,  T42 
[28] 

128  x  64  transform 
grid,  608  Mflops 

IBM  SP-2/64 

1997 

7,100 

CFD 

2  to  4 
hours 

Helicopter  rotor  free-wake  model,  high- 
order  vortex  element  and  wake  relaxation 
[38] 

Paragon  256 

1995 

7,315 

CCM 

82  sec/ 
timestep 

Particle  simulation  interacting  through  the 
standard  pair-wise  6-12  Lennar d -Jones 
potential  [40] 

50  million  particles 

8,000 

CEA 

Bottom  contour  modeling  of  shallow 
water  in  submarine  design  [20] 

8,000 

SIP 

Topological  Synthetic  Aperture  Radar 
data  processing  [20] 

Paragon  /321 

1995 

8,263 

SIP 

2-D  FFT  [41], 

200  x  1024  x 1024 
(200  Mpixels) 
images/sec 

Intel 

Paragon/321 

8,980 

SIP 

Development  of  algorithms  for  Shipboard 
Infrared  search  &  tracking  (SIRST)  [20] 

SGI 

PowerChallenge 

(R8000/150)/16 

1996 

9,510 

CFD 

3.6  hr, 

3000 

timesteps 

Large-Eddy  simulation  at  high  Reynolds 
number  [16] 

2.1  million  grid  points, 

44  Mwords 

ORNL 

Paragon/360 

9,626 

FMS 

Synthetic  forces  experiments  [42] 

5713  vehicles,  6,697 
entities 

10,000 

SIP 

Long-range  unmanned  aerial  vehicles 
(UAV)  on-board  data  processing  [20] 

Cray  T3D/128 

1995 

10,056 

CEA 

12,475  sec 

Radar  cross  section  of  perfectly 
conducting  sphere,  wave  number  20  [27] 

97x96x192  (1.7  million 
grid  points),  16.1  points 

per  wavelength 

Cray  T3D/128 

1995 

10,056 

CEA 

2,874  s 

Radar  cross  section  of  perfectly 
conducting  sphere  [27] 

128x96x92  cells  (2.4 
million  cells),  600  time 
steps 
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•  Problem  size 

CM-J/256 

1993 

10,457 

CCM 

492  CPU 
sec 

Determination  of  structure  of  Eglin-C 
molecular  system  [21] 

530  atoms,  with  1689 
distance  and  87 
dihedral  constraints 

CM-5/256 

1993 

10,457 

CCM 

7098  CPU 

sec 

Determination  of  structure  of  E.  coli  trp 
repressor  molecular  system  [21 

1504  atoms,  with  6014 
constraints 

Cray  C98 

1994 

10,625 

CWO 

~5  hrs 

Global  atmospheric  forecast.  Fleet 
Numerical  operational  run  [43] 

480  x  240  grid;  18 
vertical  layers 

Origin2000/32 

1997 

11,768 

SIP 

.205  s 

RT_STAP  benchmark  (hard)  [11] 

2.7  million  samples  per 
.161  sec 

Origin2000/32 

1997 

11,768 

CWO 

SC-MICOM,  global  ocean  forecast,  two- 
tier  communication  pattern  [44] 

Intel 

Paragon/512 

1995 

12,680 

CCM 

84  CPU 
s/timestep 

MD  simulation  of  102.4  million  particles 
using  pair-wise  6-12  Lennard  Jones 
potential  [40] 

102.4  million  atoms 

IBM  SP2/128 

1995 

13,147 

CEA 

3,304.2  s 

Radar  cross  section  of  perfectly 
conducting  sphere  [27] 

128x96x92  cells  (2.4 
million  cells),  600  time 
steps 

Maui  SP-2/128 

13,147 

FMS 

2  hours 

Synthetic  forces  experiments  [42 

5086  vehicles 

Intel  Touchstone 
Delta/512 

1993 

13,236 

CCM 

4.84 

sec/time 

step 

Molecular  dynamics  of  Si02  system  [36] 

4.2  million  atoms 

Intel  Paragon 

1995 

13,236 

SIP 

55  sec 

Correlation  processing  of  20  seconds 
worth  of  SIR-C/S-SAR  data  from  Space 
Shuttle  [45] 

NASA  Ames  SP- 
2/139 

14,057 

FMS 

2  hours 

Synthetic  forces  experiments  [42] 

5464  vehicles 

IBM  SP-2/128 

1995 

14,200 

CWO 

PCCM2,  Parallel  CCM2,  T42  [46] 

128  x  64  transform 
grid,  2.2  Gflops 

0rigin2000/40 

1997 

14,698 

CSM 

Crash  code,  PAM 

IBM  SP-2/160 

1995 

15,796 

CWO 

AGCM,  Atmospheric  General  Circulation 
Model  [47] 

144  x  88  grid  points,  9 
vertical  levels,  2.2 

Gflops 

Cray  C912 

1996 

15,875 

CSM 

23  CPU 
hours 

Water  over  C4  explosive  in  container 
above  wet  sand,  alternate  scenario: 
container  next  to  building,  finite  element 

[48] 

38,000  elements,  230 
msec  simulated  time,  16 
Mwords  memory 

Cray  C912 

1996 

15,875 

CSM 

36  hours 

water  over  C4  over  sand 

Cray  C912 

1996 

15,875 

CSM 

435  CPU 
hours 

Water  over  C4  explosive  in  container 
above  wet  sand,  building  at  a  distance 
[481 

13  million  cells,  12.5 
msec  simulated  time 

Cray  C912 

1997 

15,875 

CWO 

~  5  hrs 

Global  atmospheric  forecast.  Fleet 
Numerical  operational  run  [43] 

480  x  240  grid;  24 
vertical  layers 

Cray  C912 

1997 

15,875 

CWO 

1  hr 

Global  ocean  forecast,  Fleet  Numerical 
operational  run  [49] 

1/4  degree,  25  km 
resolution 

ORNL 

Par  agon/6  80 

16,737 

FMS 

Synthetic  forces  experiments  [42] 

10913  vehicles,  13,222 

entities 

Cray  T3D/256 

1993 

17,503 

CCM 

.0509 

CPUs/time 

step 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

100,000  atoms 

Cray  T3D/256 

1993 

17,503 

CCM 

.405 

CPUs/time 

step 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

1  million  atoms 

Cray  T3D/256 

1993 

17,503 

CCM 

1.86 

CPUs/time 

step 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

5  million  atoms 

Cray  T3D/256 

1995 

17,503 

CWO 

Global  weather  forecasting  model, 
National  Meteorological  Center,  T170 
[50] 

32  vertical  levels,  190  x 
380  grid  points,  6.1 
Gflops 

Cray  T3D/256 

1995 

17,503 

CWO 

AGCM,  Atmospheric  General  Circulation 
Model  [47] 

144  x  88  grid  points,  9 
vertical  levels,  2.5 

Gflops 
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Problem  size  i 

Cray  T3D/256 

1996 

17,503 

CWO 

105  min 

ARPS,  Advanced  Regional  Prediction 
System,  v  4.0,  fine  scale  forecast  (51] 

96  x  96  cells,  288  x 

Cray  T3D/256 

1996 

17,503 

CFD 

2  hr,  3000 
timestcps 

Large-Eddy  simulation  at  high  Reynolds 
number  [16] 

2.1  million  grid  points, 

44  Mwords 

Cray  T3D/256 

1996 

17,503 

CFD 

52,000 

CPU 
see, 130 

CPU  sec  x 
400 

timesteps 

Aerodynamics  of  missile  at  Mach  2.5,  14- 
degrees  angle  of  attack  for  laminar  and 
turbulent  viscous  effects  [34] 

944,366  nodes  and 
918,000  elements. 
4,610,378  coupled 
nonlinear  equations 
solved  every  timestep. 

CM-5/512 

1993 

20,057 

CCM 

8106  CPU 

sec 

Determination  of  structure  of  E.  coli  trp 
repressor  molecular  system  [21] 

1504  atoms,  with  6014 
constraints 

CM-5/512 

1995 

20,057 

Cfd 

15,000 

CPU  sec, 

30  CPU 

sec  per 
each  of 

500 

timesteps. 

Flare  maneuver  of  a  large  ram-air 
parachute.  Reynolds  number  »  10  million. 
Algebraic  turbulence  model  [52,53] 

469,493  nodes, 

455,520  hexahedral 
elements.  3,666,432 
equations  solved  per 
timestep. 

CM-5/512 

1995 

20,057 

CFD 

500 

timesteps 

Parafoil  with  flaps  flow  simulation.  [52] 

2,363,887  equations 
solved  at  each  timestep 

CM-5/512 

1995 

20,057 

CFD 

Parafoil  with  flaps  flow  simulation  [52]. 

2,363,887  equations 
solved  at  each  of  500 
timesteps 

CM-5/512 

1995 

20,057 

CFD 

Fighter  aircraft  at  Mach  2.0  [52] 

3-D  mesh  of  367,867 
nodes,  2,143,160 
tetrahedral  elements, 
and  1.7  million  coupled 
nonlinear  equations 
solved  per  timestep. 

CM-5/512 

1 996 

20,057 

CFD 

500 

timesteps 

Steady-state  parafoil  simulation,  10  deg 
angle  of  attack,  Reynolds  number  =  10 
million  [54] 

2.3  million  equations 
every  timestep 

CM-5/512 

1 996 

20,057 

CFD 

500 

timesteps 

Inflation  simulation  of  large  ram-air 
parachute,  Box  initially  at  10  deg  angle  of 
attack  and  velocity  at  112  ft/sec,  2 
simulated  seconds  [55] 

1,304,606  coupled 
nonlinear  equations 
solved  every  timestep. 

CM-5/512 

1996 

20,057 

CFD 

7500  CPU 
sec  =  50 
CPU  sec  x 
150 

timesteps 

Aerodynamics  of  missile  at  Mach  2.5,  14 
deg  angle  of  attack  for  laminar  and 
turbulent  viscous  effects  [34] 

763,323  nodes  and 
729,600  elements. 
3,610,964  coupled 
nonlinear  equations 
solved  in  each  of  150 
pseudo-time  steps. 

CM-5/512 

1996 

20,057 

CFD 

steady-state  parafoil  simulation.  10  degree 
angle  of  attack  [54] 

2.3  million  equations  x 
500  timesteps; 

Reynolds  number  10 
million 

CM-5/512 

1996 

20,057 

CFD 

Inflation  simulation  of  large  ram-air 
parachute.  Box  initially  at  10  degree  angle 
of  attack  and  velocity  of  112  ft/sec.  2 
seconds  simulated  [55] 

1,304,606  coupled, 
nonlinear  equations 
solved  per  each  of  500 
timesteps. 

CM-5/512 

1996 

20,05^ 

CFD 

Flare  simulation  of  large  ram-air 
parachute  [55] 

3,666,432  coupled 
nonlinear  equations 
solved  every  timestep. 

CM-5/512 

1996 

20,057 

CFD 

3-D  simulation  of  round  parachute, 
Reynolds  number  =  1  million  [56] 

CM-5/512 

1996 

20,057 

CFD 

3-D  study  of  missile  aerodynamics,  Mach 
3.5, 14  deg  angle  of  attack,  Reynolds 
number  =  14.8  million  [57]  ^ 

340,000  element  mesh, 
nonlinear  system  of 
1,750,000+  equations 
solved  every  timestep. 

CM-5/512 

1997 

20,057 

CFD 

30  hours 

Parafoil  simulation  [54] 

1  million  equations 
solved  500  times  per 
run 

CM-5/512 

20,057 

CCM 

8106  sec 
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C916 

1995 

21,12 

CSM 

900  CPU 

hours 

Hardened  structure  with  internal 
explosion,  portion  of  the  overall  structure 
and  surrounding  soil.  DYNA3D, 
nonlinear,  explicit,  FE  code.  Nonlinear 
constitutive  models  to  simulate  concrete 
&  steel  [58] 

144,257  solid  & 

168,438  trust  elements 
for  concrete  &  steel 
bars,17,858  loaded 

surfaces,  5  00,000 

DOF, 60  msec  simulated 
time 

Cray  C916 

1995 

21,12 

cwo 

CCM2,  Community  Climate  Model,  T170 
[28] 

512  x  256  transform 
grid,  2.4  Gbytes 
memory,  53.  Gfiops 

1995 

21,125 

cwo 

IFS,  Integrated  Forecasting  System,  T213 
[59] 

640  grid  points/latitude, 
134,028 

points/horizontal  layer, 

31  vertical  layers 

mmm 

1995 

21,125 

— 

ARPS,  Advanced  Regional  prediction 
System,  v  3.1  [60] 

64  x  64  x  32,  6Gflops 

Cray  C916 

1996 

21,125 

CSM 

325  CPU 

hours3  day 
continuous 
run 

Explosion  engulfing  a  set  of  buildings, 
DYNA3D  analysis  to  study  effects  on 
window  glass  &  doors  done  off-line  after 
the  blast  simulation  completed  [61] 

825  Mwords  memory 

1996 

CWO 

ARPS,  Advanced  Regional  Prediction 
System,  v  4.0,  coarse  scale  forecast  [51] 

mmm 

21,125 

explosion  engulfing  bldgs 

MHH 

1996 

CFD 

HI— 

mm 

1997 

21,125 

_ 

Bfl 

9  months 

IHif 

m 

bhi 

j^| 

SHI 

KSiiiH 

fluids 

mBBBia 

■■  Ml 

HI 

21,125 

CSM 

600  hours 

3-D  model  of  effects  of  nuclear  blast  on 
structure  [20] 

IHH 

m 

■ 

several 

hundred 

hrs 

Modeling  effects  of  complex  defensive 
structure  [20] 

Hli 

CFD 

Modeling  of  turbulent  flow  about  a 
submarine  [20] 

Avi  1 

1 

21,506 

FMS 

Synthetic  forces  experiments  [42] 

9739  vehicles 

OrigmZUUO/64 

1997 

23,488 

ChL) 

Paragon  1024 

1995 

24,520 

CWO 

PCCM2,  Parallel  CCM2,  T42  (46] 

128  x  64  transform 
grid,  2.2  Gfiops 

24,520 

CCM 

5  million  atoms 

BMPlifl 

24,520 

II 

nisi 

10  million  atoms 

Paragon/ 1024 

24,520 

FMS 

Synthetic  forces  experiments  [42] 

16995  vehicles 

Intel 

Paragon/1024 

24,520 

CCM 

50  million  atoms 

Paragon  1024 

24,520 

CCM 

200  million  particles 

Paragon  1024 

24,520 

CCM 

snoym 

400  million  particles 

ORNL 

Paragon/1024 

24,520 

FMS 

Synthetic  forces  experiments  [42] 

16606  vehicles,  20,290 
entities 
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Intel 

Paragon/1024 

Year 

1993 

j  CTP 

24,52C 

Category 

CCM 

Time 

.0282 

CPUs/ 

timestep 

Problem 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

Problem  size 

100,000  atoms 

Intel 

Paragon/1024 

1993 

24,52C 

CCM 

.199  CPUs/ 
timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

1  million  atoms 

Intel 

Paragon/ 1024 

1993 

24,520 

CCM 

.914  CPUs/ 
timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

5  million  atoms 

Intel 

Paragon/1024 

1993 

24,520 

CCM 

.961  CPUs/ 
timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

10  million  atoms 

Intel 

Paragon/1024 

1993 

24,520 

CCM 

8.54  CPUs/ 
timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

50  million  atoms 

Intel 

Paragon/1024 

1995 

24,520 

CCM 

160  CPUs/ 
timestep 

MD  simulation  of  400  million  particles 
using  pair-wise  6-12  Lennard  Jones 
potential  [40] 

400  million  atoms, 

Cray  T3D/400 

1995 

25,881 

CWO 

IFS,  Integrated  Forecasting  System,  T213 
[59] 

640  grid  points/latitude, 
134,028 

points/horizontal  layer, 

31  vertical  layers 

Mercury  Race 
(140  PowerPC 
603e  processors) 

1997 

27,113 

SIP 

Large  Mercury  System  shipped  in  1997 
[64] 

Cray  T3D/J12 

1993 

32,398 

CCM 

.0293 

CPUs/ 

timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

100,000  atoms 

Cray  T3D/512 

1993 

32,398 

CCM 

.205 

CPUs/t 

timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15 

1  million  atoms 

Cray  T3D/J12 

1993 

32,398 

CCM 

.994  CPUs/ 
timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

5  million  atoms 

Cray  T3D/512 

1993 

32,398 

CCM 

1.85  CPUs/ 
timestep 

MD  simulation  using  short-range  forces 
model  applied  to  3-D  configuration  of 
liquid  near  solid  state  point  [15] 

10  million  atoms 

Cray  T3D/512 

1995 

32,398 

CFD 

500 

timestep5 

Steady-state  parafoil  simulation.  2  deg 
angle  of  attack.  Reynolds  number  =  10 
million  [52] 

38  million  coupled 
nonlinear  equations  at 
every  pseudo-timestep. 

Cray  T3D/512 

1995 

32,398 

U*D 

Modeling  paratroopers  dropping  from 
aircraft.  Moving  grid.  Cargo  aircraft 
traveling  at  130  Knots.  High  Reynolds 
number.  Smagorinsky  turbulence  model 
[52] 

880,000  tetrahedral 
elements  for  half  of  the 
domain. 

Cray  T3D/S12 

1995 

32,398 

CPU 

Fighter  aircraft  at  Mach  2.0.  [52] 

3-D  mesh  of  185,483 
□odes  and  1,071,580 
tetrahedral  elements 

Intel  Paragon 

mid 

1990s 

44,000 

CSM 

<24  hours 

3-D  shock  physics  simulation  [12] 

6  million  cells 

Intel  Paragon 

mid 

1990s 

44,000 

CSM 

several 

restarts 

3-D  shock  physics  simulation  [12] 

20  million  cells 

Intel  Paragon 

1994 

46,000 

nuclear 

3-D  reduced  physics  simulation  oi: 
transient  dynamics  of  nuclear  weapon 
[12,65] 

ASCI  Red 

1997 

46,000 

CSM 

few 

hundred 

hours 

3-D  modeling  of  explosive  material 
impact  on  copper  plate  [12] 

1.29  billion  cells 

0rigin2000  /128 

1997 

46,928 

CEA 

Radar  cross  section  of  VFV218  aircraft 
under  2  GHz  radar  wave  [66] 

1.9  million  unknowns 

0rigin2000  /192 

1998 

70,368 

CSM 

400  hours 

armor/anti-armor,  3-D 
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mrrrm 

CTP 

■«n». 

Time 

Problem 

0rigin2000  /192 

1998 

70,368 

CFD 

months 

liBUBIIW 

■1 XiM 

100  million  cells 

250  million  cells 

ASCI  Red/1024 

mium 

2-4  million  cells 

80,000 

SIP 

Tier  2  UAV  on-board  data  processing 

122] _ 

Cray  T3E- 
900/256 

1997 

91,035 

CFD 

500 

timesteps 

2.3  million  equations 
every  rimes  tep 

Cray  T3E- 
900/256 

1997 

mm 

I 

1 

1/16  degree,  7  km 
resolution 

Cray  T3E- 
900256  nodes 

1998 

■■ 

CSM 

450,000 
□ode  hours 

lilirgsa 

ASCI  ++ 

» 

First  principles  3-D  modeling  [68] 

Source:  High-Performance  Computing,  National  Security  Applications,  and  Export  Control  Policy  at  the 
Close  of  the  20th  Century,  Seymour  E.  Goodman,  Peter  Wolcott,  and  Patrick  Homer,  (Palo  Alto,  CA: 
Stanford  University,  Center  for  International  Security  and  Arms  Control,  1998). 
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Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the  end 
of  this  appendix. 


See  comment  1 . 


See  comment  2. 


Mr.  Harold  J.  Johnson 
Director,  International  Affairs  and  Trade 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Johnson: 


THE  SECRETARY  OF  COMMERCE 

Washington,  D.C.  20230 

NOV  I  6  2000 


Thank  you  for  the  opportunity  to  review  your  draft  report.  Export  Controls:  System  for  Controlling 
Export  of  High  Performance  Computing  is  Ineffective  (GAO  code  71 1480).  We  studied  the  draft 
report  and  our  comments  and  clarifications  are  outlined  below. 

In  the  background  section,  the  GAO  refers  to  the  new  control  threshold  of  28,000  Million  Theoretical 
Operations  Per  Second  (MTOPS)  announced  by  the  Administration  in  August  2000.  Due  to  the 
FY  1998  National  Defense  Authorization  Act  (NDAA)  provisions,  this  control  threshold  will  not  take 
effect  until  February  of  2001.  The  current  licensing  threshold  is  12,500  MTOPS.  Your  report  should 
be  revised  accordingly.  Additionally,  in  the  section  entitled  ‘President’s  Justification  for  Changing 
Computer  Export  Control  Thresholds  Was  Inadequate,’  GAO  makes  several  references  to  a  1998  study 
entitled  High  Performance  Computing,  National  Security  Applications,  and  Export  Control  Policy  at 
the  Close  of  the  20th  Century  sponsored  by  Defense  and  Commerce.  GAO  states  that  we  relied  on  the 
1 998  study  to  make  our  conclusions  in  the  past  few  Presidential  Reports  to  Congress  justifying  control 
level  changes.  While  Defense  may  have  cited  this  study  in  their  portions  of  the  past  two  reports, 
Commerce  did  not  base  its  most  recent  conclusions  on  this  study.  We  use  current  assessments  of 
technology,  foreign  availability,  and  national  security  in  determining  appropriate  control  thresholds. 

Your  report  concludes  that  foreign  countries  are  now  able  to  obtain  high  performance  computing 
(HPC)  capabilities  due  to  advances  in,  and  widespread  availability  of,  clustering  technology.  We 
cannot  disagree  with  GAO  on  this  conclusion  as  it  is  a  point  we  have  made  repeatedly  to  Congress  and 
GAO  in  the  past  as  a  major  factor  affecting  export  control  policy.  We  are  pleased  that  GAO  has  taken 
note  of  the  information  we  provided  on  clustering  and  the  concerns  it  raises.  We  also  note,  as  we  have 
in  the  past,  that  some  countries  of  concern  have  indigenous  high  performance  computing  development 
initiatives,  and  this  factor  needs  to  be  taken  into  account  when  determining  the  future  course  of  HPC 
export  controls. 

We  agree  with  the  report’s  conclusion  that  the  control  parameter  for  high  performance  computers  -  - 
composite  theoretical  performance  (CTP)  as  measured  in  MTOPS  -  -  is  becoming  increasingly 
inadequate  as  a  means  to  control  high  performance  computing  capabilities.  We  are  pleased  that  GAO 
has  noted  our  efforts  on  this  front;  however,  the  tone  of  your  report  implies  that  GAO  recently 
“discovered”  the  fact  that  CTP  is  a  poor  metric.  In  reality,  for  over  a  year  now,  the  Administration  has 
been  actively  exploring  alternatives  to  CTP  that  will  allow  us  to  better  focus  our  controls  on  those 
computers  with  the  greatest  potential  military  application  without  over  regulating  our  domestic 
computer  industry  or  overburdening  our  export  control  system.  Under  Secretary  for  Export 
Administration  William  Reinsch  testified  before  Congress  on  this  subject  in  the  Fall  of  1999. 
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We  do  not  have  any  specific  comments  on  the  part  of  your  report  concerned  with  national  security 
applications  for  high  performance  computing,  since  this  topic  is  more  appropriately  addressed  by  the 
Department  of  Defense.  However,  the  President’s  report  to  Congress  clearly  states  that  military 
applications  exist  both  well  below  and  well  above  current  control  levels.  Current  control  levels  are 
largely  based  on  performance  levels  we  believe  are  controllable  in  light  of  widespread  availability  of 
computers  and  microprocessors. 

With  respect  to  the  report’s  recommendations,  we  believe  it  would  be  counterproductive  and 
unnecessary  to  convene  a  panel  of  experts  to  prepare  a  report  addressing  the  shortfalls  of  computer 
export  controls.  We  are  already  examining  this  issue  through  existing  interagency  mechanisms 
involving  the  Departments  of  Defense,  Energy  and  State.  Additionally,  we  would  also  note  that  the 
Center  for  Strategic  and  International  Studies  (CSIS)  is  currently  undertaking  a  very  similar  exercise 
that  involves  a  broad  group  of  experts  studying  the  same  issues  that  are  encompassed  in  your 
recommendation.  Creating  yet  another  group  of  experts  to  do  the  same  thing  would  be  redundant  and 
an  unnecessary  expenditure  of  taxpayer  funds. 

The  second  recommendation  in  the  report  calls  for  the  Administration  to  determine  what  counter¬ 
measures,  if  any,  are  necessary  to  respond  to  computing-related  enhancements  of  the  military  or 
proliferation  capabilities  of  countries  of  concern.  The  premise  of  our  export  controls  on  computers  is 
based  on  the  fact  that  computers  at  all  levels,  including  PCs,  have  potential  military  applications.  Our 
conclusion  is  that  the  amount  of  computing  power  needed  to  design  and  manufacture  modem  weapons, 
once  a  level  of  a  few  hundred  MTOPS  is  exceeded,  is  not  a  critical  variable.  Other  factors  -  -  software 
design,  access  to  sophisticated  manufacturing  techniques,  experience  and  test  data  -  -  are  more 
important.  For  this  reason,  our  policy  reviews  focus  less  on  potential  military  applications  and  more 
on  the  issues  of  controllability  and  technological  leadership.  The  national  security  interests  of  the 
United  States  are  best  served  by  ensuring  that  the  United  States  retains  its  technological  advantage  in 
the  design,  development,  and  production  of  microprocessors  and  computers.  As  some  50  to  60  percent 
of  revenues  for  these  sectors  are  generated  through  exports,  the  ability  to  compete  internationally  is  an 
essential  ingredient  to  the  continued  viability  of  these  industries.  Today,  no  nation  can  remain  at  the 
leading  edge  of  technology  unless  it  participates  in  the  global  market.  The  continued  generation  of 
revenues  from  exports  provides  the  needed  capital  for  research  and  development  which  keeps  these 
industries  at  the  cutting  edge  of  technology.  This,  in  turn,  ensures  that  the  U.S.  military  and  U.S. 
defense-related  industries  will  continue  to  have  access  to  the  computer  technology  which  they  need  to 
maintain  our  military  advantage  -  which  we  believe  is  the  best  countermeasure. 


Sincerely  yours. 
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The  following  are  GAO's  comments  on  the  Department  of  Commerce's 
letter  dated  November  16,  2000. 


GAO  Comments  l-  Our  report  stated  that  the  computer  licensing  level  for  tier  3  countries 

was  raised  to  28,000  MTOPS,  effective  immediately.  Commerce's  letter 
refers  to  the  28,000  MTOPS  threshold  at  which  advanced  notifications 
to  Commerce  of  computer  exports  would  be  required.  According  to  the 
President's  August  30,  2000,  letter  transmitting  his  report  justifying 
changes  to  computer  export  control  thresholds,  "the  new  level  above 
which  an  individual  license  will  be  required  for  exports  to  tier  3 
countries  is  28,000  MTOPS.  The  aforementioned  licensing  adjustments 
will  take  place  immediately."  Furthermore,  an  October  13,  2000,  rule 
amending  the  Export  Administration  Regulations  stated  that  the  upper 
licensing  level  for  computer  tier  3  countries  was  raised  from  20,000  to 
28,000  MTOPS.  In  comparison,  the  President  reported  in  the  same 
August  30,  2000,  letter  transmitting  his  report  that  he  was  establishing  a 
new  notification  level  of  28,000  MTOPS  which  cannot  take  effect  until 
180  days  after  the  Congress  receives  the  President's  Report,  that  is, 
February  2001.  Section  1211  (a)  of  the  National  Defense  Authorization 
Act  for  Fiscal  Year  1998  (P.  L.  105-85)  requires  exporters  to  notify  the 
Commerce  Department  of  any  proposed  high  performance  computer 
exports  to  countries  that  pose  a  concern  for  military  or  proliferation 
reasons  to  determine  if  these  exports  need  a  license.  Also,  the 
October  13,  2000,  rule  amendingthe  Export  Administration  Regulations 
stated  that  the  level  for  advance  notification  of  high  performance 
computer  exports  to  tier  3  countries  was  raised  from  12,500  MTOPS  to 
28,000  MTOPS,  effective  on  February  26,  2001. 

2.  We  have  clarified  the  sentences  related  to  the  1998  Defense-  and 

Commerce-sponsored  study  in  response  to  Commerce's  suggestions  by 
adding  language  to  stress  that  "DOD  officials  responsible  for  the 
sections  of  the  report  on  military  uses  of  high  performance  computers 
told  us  that  they  relied  on  the  1998  Department  of  Defense-  and 
Commerce-sponsored  study  for  support  in  the  President's  reports." 
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Washington,  DC 

Orders  by  phone: 

(202)  512-6000 
fax:  (202)  512-6061 
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